Dataset - A Standard-compliant Assessment of Beyond-eMBB QoS/QoE in 5G Networks

A Standard-compliant Assessment of Beyond-eMBB QoS/QoE in 5G Networks Giuseppe Caso∗, Mohammad Rajiullah∗, Anna Brunstrom∗, Luca De Nardis†,¨ Ozg¨ u Alay‡and Marco Neri§ ∗Department of Mathematics and Computer Science, Karlstad University, Karlstad, Sweden Email: {giuseppe.caso, mohammad.rajiullah, anna.brunstrom}@kau.se †Department of Information Engineering, Electronics and Telecommunications, Sapienza University of Rome, Rome, Italy Email: [email protected] ‡Department of Informatics, University of Oslo, Oslo, Norway Email: [email protected] §Rohde & Schwarz, Mobile Network Testing, Rome, Italy Email: [email protected] Abstract—5th Generation (5G) mobile systems are being deployed to address the Quality of Service and Experience (QoS/QoE) requirements of several use cases, including enhanced Mobile Broadband (eMBB) and Ultra-Reliable Low Latency Communication (URLLC). While eMBB performance testing inherits well-established methodologies and Key Performance Indicators (KPIs), beyond-eMBB services (i.e., URLLC and eMBBURLLC real-time applications) are often tested by adopting simplistic or in-house methodologies, which do not help towards accurate assessment and comparison. In this paper, we fill this gap by providing a detailed analysis of a methodology, recently standardized by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T), that targets systematic performance evaluations of beyond-eMBB services. The methodology relies on the definition of a QoE KPI, i.e., the interactivity score (i-score), on top of three QoS KPIs measuring service latency, stability, and continuity. To this aim, we perform a multi-service measurement campaign on two 5G networks across two cities in Sweden, during which we run a large number of tests compliant with the ITU-T methodology, and analyze the collected data. Our results empirically validate the methodology, showcasing its ability of capturing heterogeneous service characteristics and requirements, as well as the interdependencies between i-score and QoS KPIs, and the impact of different factors on QoS/QoE performance, including user mobility, connection capability, and server location. I. INTRODUCTION Standardized by the 3rd Generation Partnership Project (3GPP) in Release 15, the Fifth Generation (5G) New Radio (NR) system is designed to address the Quality of Service (QoS) and Quality of Experience (QoE) demands of enhanced Mobile Broadband (eMBB), massive Machine Type Communication (mMTC), and Ultra-Reliable Low Latency Communication (URLLC) verticals. Compared to Fourth Generation (4G) Long Term Evolution (LTE) and LTE-Advanced systems, several technological and architectural enhancements were thus conceived, including novel Radio Access Network and Core Network, and Standalone (SA) and Non-Standalone (NSA) deployments. Some of these solutions are already used in public and private 5G deployments, which exist for at least a couple of years and allow for empirical research towards further system improvement [1]. Aiming at taking into account the increasing system complexity and QoS/QoE requirement heterogeneity, performance testing methodologies should evolve along with systems and services. On the one hand, as an enhanced version of MBB services (e.g., file download/upload, web browsing, and audiovideo streaming), eMBB services inherit stable performance testing practices, which include the evaluation of the achievable throughput as the overarching test, along with other application Key Performance Indicators (KPIs), e.g., file download/upload time, web Page Load Time (PLT), video stalling time, and Mean Opinion Score (MOS). On the other hand, beyond-eMBB services, such as URLLC (e.g., industrial automation) and hybrid eMBB-URLLC (e.g., Augmented/Virtual Reality (AR/VR)), do not have older versions supported by previous system generations; the definition of new QoS/QoE KPIs is thus key for enabling systematic testing across standardization, research, and stakeholder communities [2]. Although covering different use cases, beyond-eMBB services share a common need for real-time and smooth interaction between clients and servers. Service latency,stability, and continuity thus become key aspects with higher requirements compared to eMBB, where mechanisms such as buffering and retransmissions affect client-service interaction to a lesser extent. Moreover, new QoE KPIs are needed for assessing beyond-eMBB services from an interactivity perspective. Within the above context, standardization and stakeholder communities, e.g., the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and European Telecommunications Standards Institute (ETSI), are carrying out several activities towards the definition of proper testing methodologies [3]–[5]. In parallel, the empirical research community is highlighting the importance of analyzing 5G and Beyond-5G (B5G) systems from a service perspective. However, as detailed in Section II, performance testing has been mostly focused on eMBB. Beyond-eMBB services have been recently tested by using either simplistic or proprietary methodologies, which do not enable realistic QoS/QoE assessments. More effort is thus needed for systematically assessing the performance of such services on 5G/B5G networks. In this paper, we provide a bridge across research, standardization, and stakeholder communities in the context of QoS/QoE performance testing for beyond-eMBB services on 5G/B5G systems. Our main contributions are as follows: •We detail the methodology standardized by ITU-T for testing beyond-eMBB services on 5G/B5G networks, defining the set of QoS/QoE KPIs to evaluate and the corresponding measurement procedures. •We provide an experimental assessment of such methodology via a large-scale, multi-service measurement campaign on the 5G NSA deployments of two Mobile Network Operators (MNOs), across two cities in Sweden. •We show how the methodology can be flexibly used across beyond-eMBB services. Moreover, we show its benefits over other testing approaches not reflecting real services (i.e., ping), thus enabling realistic comparative analyses across the 5G/B5G research community, which we further support by open-sourcing our dataset.1 II. BACKGROUND AND RELATED WORK Since the 5G roll-out, several empirical studies are being carried out to analyze 5G NSA and SA performance. Many works focused on throughput and latency analyses (e.g., [6]–[12]), with tests executed via iPerf2and Speedtest3 for Transmission Control Protocol (TCP) and/or User Datagram Protocol (UDP) throughput evaluation, and via Internet Control Message Protocol (ICMP) ping for latency evaluation as ping Round Trip Time (RTT). The works in [13]–[16] also provided analyses on throughput (via iPerf and/or Speedtest) and RTT (via ICMP-based traceroute). Additionally, TCP latency was analyzed in [13] and UDP latency was discussed in [15], after acknowledging that ICMP-based measurements may incur in network behaviors not reflecting real data exchange, e.g., buffering and/or filtering, as shown in previous works (e.g., [17]). These investigations also moved towards eMBB service performance testing. File download/upload performance was evaluated via download/upload time, web browsing via PLT, and video streaming via stall time and average bit rate. High-definition video telephony was tested in [15] via video throughput and frame delay, with this latter measured using an in-house methodology based on stopwatch time recording. Further application performance testing can be found in [18] where, along with iPerf-based throughput evaluation and TCP latency measurements, three different services were tested, i.e., video conferencing and streaming for the eMBB scenario, and cloud gaming as an example of beyond-eMBB service. Several KPIs were collected, including video and transmission latencies, video packet losses, and dropped frames. Video streaming was also analyzed in [19], focusing on stall time and buffer occupancy, with no latency and/or QoE evaluation. The work in [20] also addressed application performance testing. After 1Dataset available at https://doi.org/10.5281/zenodo.13987386. 2https://iperf.fr, accessed on Sept. 2024. 3https://www.speedtest.net, accessed on Sept. 2024. executing basic tests such as TCP bulk data download/upload via nuttcp4and RTT measurements via ICMP ping, the authors tested so-called 5Gkiller apps, i.e., downlink (DL)-heavy video streaming and cloud gaming (using available apps) and uplink (UL)-heavy autonomous driving and AR (using inhouse apps). Rebuffer time, bit rate, and the QoE metric defined in [21] (i.e., a linear combination of average bit rate, bit rate variation, buffering time, and video startup delay) were evaluated for video streaming, while bit rate, network latency, and percentage of dropped frames were used for cloud gaming. Moreover, end-to-end latency was evaluated for autonomous driving, while offloaded frames and object detection accuracy were measured for AR. From the above works, it is clear that the research community is increasing its effort towards real service performance testing on 5G. While eMBB testing is still predominant with stable methodologies and KPIs, beyond-eMBB evaluation has started only recently, with methodologies that do not allow for much needed comparative analyses, also due to an increasing divergence of KPIs being measured. With this paper, we move beyond the state of the art by deepening our preliminary work in [22]–[26]. In particular, we aim at detailing and empirically validating the ITU-T methodology for QoS/QoE performance testing, towards offering key insights on how beyond-eMBB services can be tested on 5G deployments in a flexible, comparable, systematic, and accurate manner. III. THE ITU-T TESTING METHODOLOGY FOR BEYOND-EMBB SERVICES Beyond-eMBB services require near real-time and uninterrupted data delivery to achieve satisfactory levels of interaction at the client side. ITU-T has thus standardized in Recommendation G.1051 (03-2023) a methodology for measuring and analyzing the QoS/QoE of these services [3]. By leveraging previous activities carried out by ETSI [4], [5], ITU-T provides the guidelines for (i) defining traffic patterns that resemble beyond-eMBB data flows, (ii) measuring a common set of QoS KPIs independently on the service under test, and (iii) modeling a QoE KPI, denoted as interactivity score (i-score [%]), as a service-specific function of the QoS KPIs. A. Defining the Traffic Patterns As a first step, the methodology requires to define a clientserver pair, in order to instantiate DL/UL data flows resembling real services between these two entities. 1) General Concepts: The client, e.g., a 5G-capable User Equipment (UE), is a packet generating unit and the server, hosting the service back-end in the Internet, is a reflecting unit. In the simplest case, each client-generated packet can be paired up with a server-reflected, same-size packet, leading to DL/UL-symmetric traffic patterns that may represent well the data flows of some services but less well the ones of other services. Therefore, aiming at also mimicking services characterized by DL/UL-asymmetric flows, the methodology 4https://www.nuttcp.net/, accessed on Sept. 2024. requires the server to be able to reflect packets of different sizes. By doing so, DL/UL-asymmetric traffic patterns can be created, with bit rates depending on both packet size (potentially different between client and server) and packet generation rate (the same between client and server, so that each client packet is still paired up with one server packet). During a test, packet size and rate can change to emulate different service phases, the order of which follows a realistic service delivery profile, e.g., initialization and trailing should be the first and last phases, with intermediate phases of different bit rates representing different interaction situations. As a guideline, ITU-T highlights that the implemented patterns should represent similar applications in the same service class, with this latter being, e.g., cloud gaming or industrial automation. Patterns should thus be created after in-depth analyses, e.g., in terms of DL/UL bit rates, packet sizes, and packet generation rates, of traces collected on real applications during different usage phases. 2) Protocol Implementation: Due to the focus on realtime services and aiming at avoiding uncontrolled traffic (e.g., retransmissions), the ITU-T methodology specifies User Datagram Protocol (UDP) as the transport protocol. On top of UDP, it is recommended to use Two-Way Active Measurement Protocol (TWAMP), standardized by the Internet Engineering Task Force (IETF) in the Request for Comments (RFC) 5357 [27]. In its initial version, TWAMP enables same-size packet generation/reflection. Asymmetric packet reflection is foreseen in later TWAMP versions [28], therefore, ITU-T currently allows for implementation-specific solutions. UDP packets could be fragmented if their size exceeds the maximum transmission unit allowed on the client-server network path. If so, fragments are exchanged sequentially and the receiving side reassembles them back into the original UDP packets. 3) Practical Examples: We now describe the methodologycompliant patterns for the three services considered in our analyses on 5G NSA networks, as detailed in later sections. eGaming real-time: This is a DL/UL-symmetric pattern that emulates the interaction of a user with a typical online multiplayer gaming application. During a test of 10 seconds, phases with low-to-medium data rates are generated assuming that only status information is exchanged between client and server, with video processing performed at the user end. A starting phase (bit rate of 100 kb/s) is followed by a mediuminteractivity phase (bit rate of 300 kb/s), a high-interactivity phase (bit rate of 1Mb/s), and two final phases (bit rate of 300 and 100 kb/s). The bit rates are obtained by generating 125 to 1250 100-byte packets per second. AR/VR Cloud Gaming: This is a DL/UL-asymmetric traffic pattern derived by analyzing data flows exchanged by users with cloud gaming platforms. During a test of 10 seconds, the pattern emulates cloud services where the client constantly sends status information (e.g., head orientation and pose) to a server, using a low bit rate of 0.25 Mb/s. The server processes client info and sends the corresponding high definition video (up to 1080p and 60 frames per second) back to it in three consecutive phases of medium-high data rate (2Mb/s, 5Mb/s, and 2Mb/s). The bit rates are obtained by using 540-byte packets in UL and 5460/10920-byte packets in DL, with 60 packets per second generated in both directions. Industry 4.0(I4.0)Process Automation: This is a DL/ULsymmetric pattern based on URLLC services in the I4.0 category, as described in 3GPP Technical Report (TR) 22.804 [29]. It is assumed that sensors-collected data are fed to a controller deciding on how to configure actuators. The data rate requirement (0.5Mb/s) is lower compared to previous patterns, but the test duration is longer (30 seconds), in order to accurately assess service continuity (described along with latency and stability in the next section, together with the corresponding QoS KPIs). The bit rate is obtained by using 100-byte packets with 625 packets generated per second. B. Measuring the QoS KPIs Once traffic patterns are defined, the methodology requires to quantify latency, stability, and continuity by measuring three QoS KPIs, in the same way across patterns. 1) Latency: The quantification of service latency is executed by measuring the RTT of each pair of packets (in case of fragmentation, RTT is evaluated on the last fragment). At the end of a test, RTT statistics for the data flow can be evaluated, e.g., as x-th percentiles and/or standard deviation. 2) Stability: The quantification of service stability is executed by evaluating the latency variation, i.e., jitter, experienced by packets over the client-server network. This is measured via the definition of Packet Delay Variation (PDV) given in IETF RFC 5481 [30], i.e., the difference between the RTT of a packet and the minimum RTT across all packets (in case of fragmentation, PDV is evaluated on the last fragment). PDV statistics for the data flow can be evaluated at the end of a test, e.g., as x-th percentiles and/or standard deviation. 3) Continuity: The quantification of service continuity is executed by evaluating the so-called Packet Loss Rate (PLR), i.e., the ratio between disqualified packets and the number of packets scheduled by the client. Packets are considered disqualified if not usable for a real-time service, and thus include not sent packets (due to UL congestion), lost packets (due to DL congestion), and discarded packets, i.e., packets received by the client after a latency budget (in case of fragmentation, a packet is disqualified if a fragment is disqualified). Considering the three patterns above, the latency budget is 100 ms for eGaming real-time and AR/VR Cloud Gaming, to reflect the 3GPP requirement for the application class both services can be mapped to, i.e., 5G QoS Identifier (5QI) Class 3[31]. A latency budget of 20 ms is considered for I4.0 Process Automation, as per 5QI Class 82 [29], [31]. C. Modeling the QoE KPI Once service latency, stability, and continuity are quantified via the corresponding QoS KPIs, the methodology provides a way to define a single QoE KPI, the i-score, which incorporates the three QoS KPIs to quantify the client-perceived service responsiveness [3]–[5]. While the i-score model is TABLE I i-score MODEL PARAMETERS [3]. Traffic Pattern Model Parameters fmax a b u v eGaming real-time 100 61 14 120 4 AR/VR Cloud Gaming I4.0Process Automation 15 2 10 2500 service-agnostic, the model parameters are service-specific, so to reflect the difference in terms of service requirements. The definition of i-score reflects the need for new ways of evaluating the QoE of beyond-eMBB services, towards also including services not involving humans, where human-centered MOS assessments are evidently not an option. As a result, a more general way to assess QoE performance was introduced, in order to identify environment and system conditions under which service requirements cannot be fulfilled. When deriving the i-score, it is assumed that service interactivity has a monotonous inverse dependency on service latency, with saturation areas at low and high latency values since no interactivity changes are perceived at these extreme ranges. Therefore, a logistic function with service-specific parameters fmax,a, and bis used to model the dependency of service interactivity on latency, so that RTTs from nondisqualified packets can be transformed into values between 0% and fmax. Assuming Nnon-disqualified packets collected during a test, the RTT-dependent term of the i-score model, denoted scoreRTT, is evaluated as follows: scoreRTT =1 N N X n=1 fmax f0"1−1 1 + e−(RTTn−a) b#, where f0= 1 −1 1 + ea b . (1) PDV and PLR are accounted for by defining scorePDV and scorePLR, with service-specific parameters uand v, as follows: (scorePDV = max(0,1−σPDV u) scorePLR = max(0,1−v×PLR) (2) where σPDV is the standard deviation of the PDV over nondisqualified packets. The i-score is then calculated as follows: i-score =scoreRTT ×scorePDV ×scorePLR.(3) Table I reports the i-score model parameters for the services described above. We highlight how, in order to reflect the 3GPP mapping to the same application class, the i-score model for e-Gaming real-time and AR/VR Cloud Gaming uses the same parameters. These services only differ in their traffic patterns. Moreover, Figure 1 shows examples of the i-score model as a function of the underlying QoS KPIs, particularly showcasing how different parameters (e.g., between e-Gaming real-time/AR/VR Cloud Gaming and I4.0Process Automation) affect the i-score curves, thus reflecting different service requirements. For example, I4.0Process Automation services require extremely low RTT in order to achieve a high iscore, and are more affected by stability and/or continuity degradation compared to gaming and AR/VR applications. 0 20 40 60 80 100 RTT [ms] 0 10 20 30 40 50 60 70 80 90 100 i-score [%] eGaming and AR/VR ( PDV = 0 ms, PLR = 0) eGaming and AR/VR ( PDV = 5 ms, PLR = 10-4 ) eGaming and AR/VR ( PDV = 5 ms, PLR = 10-1 ) I4.0 Process Automation ( PDV = 0 ms, PLR = 0) I4.0 Process Automation ( PDV = 5 ms, PLR = 10-4) I4.0 Process Automation ( PDV = 5 ms, PLR = 10-1) Fig. 1. Examples of i-score models for e-Gaming real-time,AR/VR Cloud Gaming, and I4.0Process Automation services. Curves are derived by using the parameters in Table I, and for different values of σPDV and PLR. IV. MEASUREMENT CAMPAIGN AND DATASET In this section, we describe the setup and measurements used for our empirical assessment of the ITU-T methodology. We executed measurements in Stockholm and Karlstad in June-November 2023, by using two 5G-capable UEs (Samsung S20), embedded with the Rohde & Schwarz (R&S) Qualipoc Android app5and connected to an Intel Windows PC running ROMES, a R&S software enabling measurement configuration, real-time inspection, and data exporting. We organized our measurements in Indoor Static (IS), Outdoor Walking (OW), and Outdoor Driving (OD) campaigns. Overall, we covered 7IS locations (4/3in Stockholm/Karlstad), along with 6OW (3/3) and 3OD areas (2/1). In each location/area, we repeated campaigns over different days and times of the day. Including repetitions, our collection in the OD scenario covered about 200 km across the two cities. We used the UEs to execute service performance tests on the 5G NSA networks of two Swedish MNOs, referred in the following to as OP1and OP2. Both deployments are in the mid band (Band n78,3.3-3.8GHz) and use Time Division Duplexing (TDD). Moreover, we configured our UEs to alternatively operate in two connectivity modes, i.e., LTE Only, which forced the UEs to only expose 4G capabilities, and 5G NSA, which enabled the UEs to expose 5G capabilities, so that the MNOs could connect them to 5G and 4G Physical Cell Identifiers (PCIs) for exploiting 4G-5G dual connectivity. We performed the so-called Interactivity Test6, a routine available in R&S Qualipoc that makes it possible to run tests following the ITU-T methodology. We run tests with eGaming real-time,AR/VR Cloud Gaming, and I4.0Process Automation patterns, after installing the service back-end in four different servers, two in Sweden (Stockholm/Karlstad, SE-S/SE-K), one in Switzerland (Zurich, CH) and one in Italy (Rome, IT). Tests were repeated several times for each campaign, and a single test repetition is referred to as a session in the following. We also performed 10-second ICMP ping sessions (ten 32byte pings per session) towards SE-Sand SE-Kservers, in the same locations and conditions of the Interactivity tests. 5https://tinyurl.com/y6c6z9eu, accessed on Sept. 2024. 6https://tinyurl.com/bde5mjtf, accessed on Sept. 2024. In 5G NSA mode, we ran about 7900 Interactivity sessions for each MNO, across scenarios, patterns, and servers, with a higher percentage of sessions for the IS scenario, eGaming real-time and AR/VR Cloud Gaming patterns, and SE-S/SE-K servers. Towards these two servers, we ran about 150 ICMP ping sessions for each MNO, across scenarios. In LTE Only mode, we ran about 20% of the sessions executed in 5G NSA. The collected dataset includes information on QoS/QoE performance observed during each session, i.e., applicationlayer DL/UL throughput, RTT, PDV, PLR, and i-score of the Interactivity tests, and RTT of the ICMP pings. Additionally, it includes coverage indicators (e.g., Reference Signal Received Power and Signal-to-Interference plus Noise Ratio of the 4G/5G PCIs at which the UEs were connected) and resource allocation information (e.g., Modulation and Coding Scheme). V. RESULTS In this section, we analyze the data collected during our measurements, aiming at providing an in-depth, data-driven characterization of the ITU-T methodology. A. Comparison between ICMP ping and Interactivity tests We start our analysis by showing the performance difference observed when running ICMP ping vs. Interactivity tests over the 5G NSA networks of OP1and OP2(UE mode is 5G NSA). We focus on SE-Sand SE-Kservers and, for each ping, eGaming real-time, and AR/VR Cloud Gaming session, we evaluate the median RTT, PDV, and PLR. For the Interactivity tests, these values are provided by R&S Qualipoc; for the ping case, we evaluate them by applying the ITU-T methodology rationale, i.e., median RTT and PDV are evaluated on pings with RTT ≤100 ms, while PLR accounts for pings with RTT >100 ms. Figure 2a shows that, for both MNOs, ping performance is different from the Interactivity ones. For OP1, ping RTT is consistently higher in median than Interactivity RTT for both patterns, despite the smaller packet size. However, the RTT increases when moving from eGaming real-time to AR/VR Cloud Gaming, due to the larger DL packets in the latter. This result hints at possible different strategies applied by OP1in handling ICMP and UDP traffic, with a sort of low priority applied to the former. Moreover, while ping PDV is in line with Interactivity PDV, ping PLR underestimates Interactivity PLR. For OP2, ping RTT, PDV and PLR are consistently lower than the Interactivity ones, with increasing trends across the two patterns. In line with previous analyses [17], our results show that ping should not be used to infer real service performance, also due to possible different strategies that networks may apply for handling protocols like ICMP. B. Performance of i-score across services We now assess i-score performance measured during the Interactivity tests with the three traffic patterns previously described. Figure 2b shows the empirical cumulative distribution function (ECDF) of the i-score for eGaming real-time, AR/VR Cloud Gaming, and I4.0Process Automation, after combining MNOs, scenarios, and servers (UE mode is 5G 0 50 100 RTT [ms] Ping to SE-K eGaming to SE-K AR/VR to SE-K Ping to SE-S eGaming to SE-S AR/VR to SE-S 0 20 40 PDV [ms] OP1OP2 MNO 0 5 10 PLR [%] (a) 0 20 40 60 80 100 i-score [%] 0 0.2 0.4 0.6 0.8 1 Empirical Cumulative Distribution Function eGaming real-time AR/VR Cloud Gaming I4.0 Process Automation (b) Fig. 2. (a) RTT, PDV, and PLR measured for OP1and OP2during ICMP ping and Interactivity test sessions towards SE-Sand SE-Kservers; (b) iscore ECDF for eGaming real-time,AR/VR Cloud Gaming, and I4.0Process Automation services (MNOs and servers combined). In (a)(b), IS, OW, and OD scenarios are combined, and UE mode is 5G NSA. NSA). The results show significant performance differences; although eGaming real-time and AR/VR Cloud Gaming have the same i-score model parameters (Table I), their results are quite different, with the second pattern facing more challenges in achieving high performance, as a direct result of its heavier DL traffic. In 50% of sessions, the i-score is lower than 60% for AR/VR Cloud Gaming and 70% for eGaming real-time, with eGaming real-time thus better supported by the 5G NSA deployments of OP1and OP2. Moreover, the i-score of I4.0 Process Automation is always 0%, thus showing that its strict requirements cannot be satisfied on current 5G NSA networks and likely require 5G SA deployments. For this traffic pattern, we mostly observe a median PLR close to 100%. For SE-S and SE-Kservers, due to shorter client-server distances, we observe sporadic sessions with PLR as low as 15% (SE-S) and 50% (SE-K), which is however not sufficient due to the tight PLR requirements for this service, as shown in Figure 1. C. From QoS to QoE through the i-score model We continue our analysis by providing more insights on the interdependencies between RTT, PDV, PLR, and i-score. Figure 3 shows the scatter plots between each pair of QoS (RTT, PDV, and PLR) and QoE (i-score) KPIs, in the median values measured during eGaming real-time and AR/VR Cloud Gaming sessions, across MNOs, scenarios, and servers (UE mode is 5G NSA). The sub-figures on the main diagonal also show the histogram of the values measured for each KPI. Several interesting insights can be derived from this figure. First, the RTT vs. i-score plot shows a noisy logistic function, resembling Figure 1 and according to (1). The spreading of i-score values highlights how PDV and PLR affect the iscore on top of RTT, following (3). We also see that the RTT distribution of AR/VR Cloud Gaming is slightly shifted towards higher values compared to the eGaming real-time case, in alignment with Figure 2a. Moreover, high PDV/PLR result in low i-score values, while RTT, PDV, and PLR are positively correlated. For AR/VR Cloud Gaming, we observe a cluster of points with PLR close to 100% and thus i-score equal to 0%. For this result, we provide more details in Figure 4, where we further split our measurements across servers to assess their impact on performance. Figure 4 presents parallel coordinate plots showing the median RTT, PDV, PLR, and i-score Fig. 3. Scatter plots between RTT, PDV, PLR, and i-score measured during eGaming real-time (blue) and AR/VR Cloud Gaming (red) tests. MNOs, scenarios, and servers are combined, and UE mode is 5G NSA. Figures on the main diagonal show the histogram of the values measured for each KPI. RTT [ms] PDV [ms] PLR [%] DL Throughput [Mb/s] UL Throughput [Mb/s] i-score [%] 0 20 40 60 80 100 Coordinate Value eGaming real-time SE-S SE-K CH IT 0 0.5 1 1.5 (a) RTT [ms] PDV [ms] PLR [%] DL Throughput [Mb/s] UL Throughput [Mb/s] i-score [%] 0 20 40 60 80 100 Coordinate Value AR/VR Cloud Gaming SE-S SE-K CH IT 0 2 4 (b) Fig. 4. Parallel coordinate plots for eGaming real-time (a) and AR/VR Cloud Gaming (b), across different servers. Plots show the median RTT, PDV, PLR, DL/UL throughput, and i-score across sessions (UE mode is 5G NSA). Zoomin figures are provided for better visualizing PLR and DL/UL throughput. measured across eGaming real-time (Figure 4a) and AR/VR Cloud Gaming (Figure 4b) sessions. Additionally, we report the median DL and UL throughput measured across sessions. Figure 4a highlights that the main factor affecting eGaming real-time performance is the RTT: across all four servers, RTT increases as function of the client-server distance and, in turn, the corresponding i-score decreases. We also see that, for all servers, the DL/UL-symmetric data rate of this service is always sustained, with median value slightly below 0.5Mb/s, in line with the traffic characteristics described in Section III-A3. Figure 4b shows similar results across servers and highlights the DL/UL-asymmetric data rate of AR/VR Cloud Gaming. However, a peculiar behavior is evident for the IT server: this is characterized by a higher PLR as the result of a lower median DL throughput, which is expected to be around 4Mb/s (as for the other servers and in line with traffic characteristics) while it is close to zero. During AR/VR Cloud Gaming sessions with the IT server, we registered a large number of DL lost packets, indicating server limitation and/or DL congestion eventually captured in the i-score KPI. D. Performance benchmarking via i-score We finally showcase how the ITU-T methodology can be used for performance benchmarking. We further split our measurements across MNOs and scenarios, and also analyze the performance observed with UE connectivity mode equal to LTE Only. Figure 5 shows the i-score performance measured during eGaming real-time sessions, for OP1(left) and OP2 (right) across all servers. For both MNOs, we compare performance as a function of the UE connectivity mode. Note that we focus on eGaming real-time and IS but our observations also apply to AR/VR Cloud Gaming and OW, for which we do not report figures due to space limitation (we did not execute LTE Only sessions in the OD case). Results show that, in all cases, the i-score consistently decreases as a function of client-server distance. Latency/reliability-sensitive beyondeMBB services thus require dedicated network architectures for a full support, e.g., including edge deployments for reducing user-client distance and latency. Moreover, for OP1, 5G NSA always outperforms LTE Only in the median values, thus showcasing the beneficial effect of 5G NSA systems for beyond-eMBB support. OP2mostly shows the opposite trend, with no evident 5G NSA benefit, and LTE Only actually outperforming 5G NSA in a few cases (e.g., with SE-Sserver). For both MNOs, we also see larger variability for 5G NSA, ultimately highlighting that 5G NSA can indeed bring benefits for beyond-eMBB service support, but early deployments as 5G NSA LTE Only 0 20 40 60 80 100 i-score [%] IS - OP1 eGaming to SE-S eGaming to SE-K eGaming to CH eGaming to IT 5G NSA LTE Only Connectivity Mode 0 20 40 60 80 100 IS - OP2 Fig. 5. i-score measured during eGaming real-time sessions, for OP1(left) and OP2(right), in the IS scenario, across all servers, and as a function of the UE connectivity mode (5G NSA vs. LTE Only). the ones currently available may lead to instability, ultimately requiring dedicated solutions for performance enhancement. VI. CONCLUSION In this paper, we provide a data-driven analysis of the ITU-T methodology for beyond-eMBB performance testing on 5G/B5G, which overcomes accuracy and comparability limitations caused by simplistic or in-house methodologies. By exploiting measurements on two 5G NSA networks, we validate the ability of the i-score – the QoE KPI modeled on top of RTT, PDV, and PLR – of capturing service characteristics and requirements by incorporating three QoS KPIs. Our results, spanning across mobility scenarios, connection capabilities, and server locations, highlight how the methodology can be used for reliable and accurate assessments of beyond-eMBB services on 5G/B5G. In future work, we will investigate the dependencies between i-score and low-layer parameters related to coverage and configurations, towards understanding how such factors affect performance, eventually deriving prediction schemes and solutions to overcome performance bottlenecks. We also plan to extend the methodology by including additional service classes and traffic patterns, e.g., in the context of vehicle-to-everything applications. ACKNOWLEDGMENTS The work by G.C., M.R., and A.B. is supported by the Knowledge Foundation of Sweden and by the EU HE programme (agreement No. 101139172, 6G-PATH). The work by L.D.N. is supported by the EU under the Italian National Recovery and Resilience Plan (RESTART). The work by ¨ O.A. is supported by the EU HE programme (agreement No. 101096452, IMAGINE-B5G). REFERENCES [1] G. Caso et al., “Experimentation in 5G and beyond Networks: State of the Art and the Way Forward,” Sensors, vol. 23, no. 24, p. 9671, 2023. [2] A. D´ ıaz Zayas et al., “A modular experimentation methodology for 5G deployments: The 5GENESIS approach,” Sensors, vol. 20, no. 22, 2020. [3] ITU-T, “Latency measurement and interactivity scoring under real application data traffic patterns,” Recommendation ITU-T G.1051, 2023. [4] ETSI, “Speech and multimedia Transmission Quality (STQ); QoS parameters and test scenarios for assessing network capabilities in 5G performance measurements,” ETSI TR 103 702 v1.1.1, 2020. [5] ——, “Speech and multimedia Transmission Quality (STQ); Best practices for robust network QoS benchmark testing and scoring,” ETSI TR 103 559 v1.2.1, 2023. [6] A. Narayanan et al., “Lumos5G: Mapping and Predicting Commercial mmwave 5G Throughput,” in Proc. of the ACM Internet Measurement Conference, 2020, pp. 176–193. [7] R. Fezeu et al., “An in-depth measurement analysis of 5G mmwave PHY latency and its impact on end-to-end delay,” in International Conference on Passive and Active Network Measurement, 2023, pp. 284–312. [8] C. Fiandrino et al., “A study on 5G performance and fast conditional handover for public transit systems,” Computer Communications, vol. 209, pp. 499–511, 2023. [9] A. A. El-Saleh et al., “Measurements and Analyses of 4G/5G Mobile Broadband Networks: An Overview and a Case Study,” Wireless Communications and Mobile Computing, vol. 2023, 2023. [10] M. I. Rochman et al., “A comprehensive analysis of the coverage and performance of 4G and 5G deployments,” Computer Networks, vol. 237, p. 110060, 2023. [11] W. Ye et al., “A Closer Look at Stand-Alone 5G Deployments from the UE Perspective,” in Int’l Mediterranean Conference on Communications and Networking. IEEE, 2023, pp. 86–91. [12] Y. Liu and C. Peng, “A Close Look at 5G in the Wild: Unrealized Potentials and Implications,” in IEEE Int’l Conference on Computer Communications, 2023. [13] A. Narayanan et al., “A First Look at Commercial 5G Performance on Smartphones,” in Proc. of The Web Conference, 2020, pp. 894–905. [14] ——, “A Variegated Look at 5G in the Wild: Performance, Power, and QoE Implications,” in Proc. of the ACM SIGCOMM Conference, 2021, pp. 610–625. [15] D. Xu et al., “Understanding Operational 5G: A First Measurement Study on its Coverage, Performance and Energy Consumption,” in Proc. of the ACM SIGCOMM Conference, 2020, pp. 479–494. [16] R. A. Fezeu et al., “Mid-Band 5G: A Measurement Study in Europe and US,” arXiv preprint arXiv:2310.11000, 2023. [17] C. Pelsser et al., “From Paris to Tokyo: On the suitability of ping to measure latency,” in Proceedings of the ACM Internet Measurement Conference, 2013, pp. 427–432. [18] A. Hassan et al., “Vivisecting mobility management in 5G cellular networks,” in Proc. of the ACM SIGCOMM Conference, 2022. [19] E. Ramadan et al., “Case for 5G-aware Video Streaming Applications,” in Proc. of the ACM SIGCOMM Conference, Workshop on 5G Measurements, Modeling, and Use Cases, 2021, pp. 27–34. [20] M. Ghoshal et al., “Performance of Cellular Networks on the Wheels,” in Proc. of the ACM Internet Measurement Conference, 2023, pp. 678– 695. [21] X. Yin et al., “A control-theoretic approach for dynamic adaptive video streaming over HTTP,” in Proc. of the ACM SIGCOMM Conference, 2015, pp. 325–338. [22] K. Kousias et al., “Coverage and Performance Analysis of 5G NonStandalone Deployments,” in Proc. of the ACM MobiCom Workshop on Wireless Network Testbeds, Experimental Evaluation and Characterization, 2022. [23] ——, “Implications of handover events in commercial 5G nonstandalone deployments in Rome,” in Proc. of the ACM SIGCOMM Conference, Workshop on 5G and Beyond Network Measurements, Modeling, and Use Cases, 2022, pp. 22–27. [24] ——, “A Large-Scale Dataset of 4G, NB-IoT, and 5G Non-Standalone Network Measurements,” IEEE Communications Magazine, 2023. [25] G. Caso et al., “An Initial Look into the Performance Evolution of 5G Non-Standalone Networks,” in Proc. of the Network Traffic Measurement and Analysis Conference. IEEE, 2023, pp. 1–4. [26] K. Kousias et al., “Empirical performance analysis and ML-based modeling of 5G non-standalone networks,” Computer Networks, vol. 241, p. 110207, 2024. [27] IETF, “A Two-Way Active Measurement Protocol (TWAMP),” IETF RFC 5357, 2008. [28] ——, “Two-Way Active Measurement Protocol (TWAMP) Reflect Octets and Symmetrical Size Features,” IETF RFC 6038, 2010. [29] 3GPP, “Study on Communication for Automation in Vertical domains (CAV),” Technical Report (TR) 22.804, Oct. 2017. [30] IETF, “Packet Delay Variation Applicability Statement,” IETF RFC 5481, 2009. [31] 3GPP, “5G; System Architecture for the 5G System (Rel-15),” Technical Specification (TS) 23.501, 2018.