Optimizing H.265 Real-Time Streaming

Lukas Fuchs, BSc Optimizing H.265 Real-Time Streaming MASTER’S THESIS to achieve the university degree of Diplom-Ingenieur Master’s degree programme Computer Science submitted to Graz University of Technology Supervisor Priv.-Doz. Dipl.-Ing. Dr.techn. Clemens Arth Institute of Computer Graphics and Vision Advisor Dipl.-Ing. Dr.techn. Philipp Fleck Institute of Computer Graphics and Vision Graz, Austria, September 2025 Abstract In this thesis, we tried to optimize the current State-of-the-Art H.265 live streaming especially for panoramic video captures. The idea was to utilize the tiling technique used in the H.265 codec, which allows the video to be divided into independent regions. By transmitting only the currently seen part of the panoramic video in high quality, and reducing the quality the further away the Tiles are from the current viewport, we can reduce the bandwidth requirements while still maintaining a seamless, low latency, high-quality live experience. To realize this, we set up a server environment that is capable of receiving a H.265 encoded panoramic live video stream, which is then converted into multiple quality levels while splitting the entire panoramic video into individual regions, which can be downloaded separately. To obtain those tiles in downloadable format, we utilized the DASH protocol, which is capable of creating a media presentation description file that includes all the information about the position, time, and quality level of each tile and can be interpreted by a client. For the client side, we decided to use the Unity Engine to create an Android application which can be run on VR devices as well as smartphones. There we implemented a tile selection method that can divide the tiles into four different groups depending on their position relative to the current view of the user. With that information, we download the correct files from the server and recreate a playable video containing multiple quality-level tiles, which drastically reduces the network requirements. In addition, we implemented a custom video player using the native Android Media library to ensure seamless video playback without stutter or delay and provide the user with the best possible viewing experience. iii Kurzfassung In dieser Arbeit haben wir versucht, den aktuellen Stand der Technik im Bereich des H.265 Livestreaming f¨ur Panorama Videoaufnahmen zu optimieren. Die Grundidee bestand darin, die enthaltene Tiling Technik des H.265 Codec zu nutzen, die es erm¨oglicht das Video in unabh¨angige Regionen zu unterteilen. Durch die ¨ Ubertragung der Bildbereiche in hoher Qualit¨at, welche sich im Sichtfeld des Benutzers befinden, und die Reduzierung der Qualit¨at, desto weiter die einzelnen Bildbereiche vom Sichtfeld entfernt sind, lassen sich die Netzwerk Anforderungen verringern, w¨ahrend gleichzeitig ein nahtloses, latenzfreies, hoch qualitatives Live Erlebnis gew¨ahrleistet wird. Um dies umzusetzen, haben wir eine Serverumgebung eingerichtet, welche in der Lage ist, einen H.265 kodierten Panorama Livestream zu empfangen. Dieser wird dann in mehreren Qualit¨atsstufen konvertiert, w¨ahrend das ganze Bild in unabh¨angige Regionen unterteilt wird, die separat heruntergeladen werden k¨onnen. Um diese in einem geeigneten Format bereitstellen zu k¨onnen, haben wir das DASH-Protokoll eingesetzt, welches eine Media Presentation Description Datei erstellt, die s¨amtliche Informationen ¨uber Position, Zeit, und Qualit¨atsstufe der einzelnen Bildbereiche enth¨alt und von einem Endger¨at interpretiert werden kann. F¨ur die Client Seite haben wir uns entschieden die Unity Engine zu verwenden, um eine Android Anwendung zu entwickeln welche auf VR Ger¨aten als auch auf Smartphones ausgef¨uhrt werden kann. Dort implementierten wir eine Tile Auswahl, die anhand der relativen Position zur derzeitigen Blickrichtung, die Tiles in vier Kategorien unterteilt. Mit dieser Information werden die passenden Dateien vom Server geladen und zu einem funktionierendem vollst¨andigen Panoramavideo zusammengesetzt, bestehend aus verschiedenen Qualit¨atsstufen, was die Netzwerkanforderungen stark reduziert. Zus¨atzlich haben wir mithilfe der Android Media Bibliothek einen maßgeschneiderten Videoplayer entwickelt, welcher eine unterbrechungsfreie Wiedergabe ohne Ruckeln und Verz¨ogerungen erm¨oglicht, um dem Endbenutzer das bestm¨ogliche Erlebnis zu bieten. v Affidavit I declare that I have au thored this thesis independently, that I have not used other than the declared sources/resources, and that I have explicitly indicated all material which has been quoted either literally or by content from the sources 'used. The text document uploaded to TUGRAZonline is identical to the present master's thesis dissertation. Date Signature Acknowledgments Firstly, I would like to thank my supervisor, Priv.-Doz. Dipl.-Ing. Dr-techn. Clemens Arth, who gave me the opportunity to create this thesis, and for his continuous help and valuable feedback throughout the entire process. I am also grateful to my advisor, Dipl.- Ing. Dr.techn. Phillip Fleck for always being available and providing guidance during the development of this work. Secondly, I would like to thank my family for their continuous support throughout my years of study, for giving me the strength to achieve my goals, and for making it possible for me to pursue my education at the university. Special thanks go to my girlfriend, who supported me all the time and motivated me to continue and reach my goal, and to my friends, on whom I could always count. ix xvi LIST OF FIGURES 4.8 Comparison of function durations tested on the Meta Quest 3. . . . . . . . 39 4.9 Timing diagram of frame generation and function calls tested on the Meta Quest 3 with YUV packaging to an RGBA buffer utilizing SIMD instructions and converted on the GPU to accurate pixel values. . . . . . . . . . . 40 5.1 Usage statistics of the server while processing a 20mb/s 4K video. . . . . . 45 5.2 Quality comparison of the tiled DASH 360°4K video and non-tiled version with the same download rate per device. . . . . . . . . . . . . . . . . . . . . 48 5.3 Download rate of tiled 20Mb/s video with example tiled video. . . . . . . . 49 5.4 Timing diagram of function calls on a 4K video at 30 fps with different color conversion techniques, tested on Meta Quest 2. The color of the FCO indicates, whether the frame was generated in the correct time shown in green, red, if it was generated too late, and purple, if a frame needed to be dropped....................................... 51 5.5 Timing diagram of function calls on a 4K video at 30 fps with different color conversion techniques, tested on Meta Quest 3. The color of the FCO indicates, whether the frame was generated in the correct time shown in green, or red, if it was generated too late. . . . . . . . . . . . . . . . . . . . 51 List of Tables 5.1 Test server setup specification. . . . . . . . . . . . . . . . . . . . . . . . . . 44 5.2 Influence of GOP and Frame Parallelism settings on CPU and Memory usage. Tested with a resolution of 3840 ×1920, a framerate of 30 FPS, an upload rate of 20Mb/s, a tile layout of 8 ×4 and two tile qualities with 20Mb/sand1Mb/s. ............................... 45 5.3 The effect of different tile qualities and different framerates on CPU and Memory usage. Tested with a resolution of 3840 ×1920, a tile layout of 8×4, a GOP size of 8 and Frame Parallelism of 2 . . . . . . . . . . . . . . 45 5.4 The effect of different tile layouts on CPU and Memory usage. Tested with a resolution of 3840×1920, a framerate of 30 FPS, an upload rate of 20Mb/s, a GOP size of 8, Frame Parallelism of 2 and two tile qualities with 20Mb/s and1Mb/s. .................................... 46 5.5 The effect of different resolutions on CPU and Memory usage. Tested with a GOP size of 8 and Frame Parallelism of 2 . . . . . . . . . . . . . . . . . . 46 5.6 Client download rates with different quality and tile configurations, a fixed resolution of 3840 ×1920 and an upload rate of 20Mb/s. The tile visibility indicates the number of fully visible, partial visible, surrounding and non visible tiles selected by the client, and the tile quality specifies how many tiles are selected at each quality level. . . . . . . . . . . . . . . . . . . . . . 50 5.7 Abbreviation description for timing diagrams of different conversion techniques........................................ 51 xvii xviii LIST OF TABLES 5.8 Average frame conversion times and Decoder times for processing the first frame of a segment after 120 frames of different conversion techniques, tested on Meta Quest 3, Meta Quest 2 and Samsung A53. Conversion stands for directly converting the YUV data to RGBA on the CPU, Pointer stands for packaging the YUV data to the RGBA buffer and convertion done on theGPU. ..................................... 53 5.9 Showing the Latency differences for one, two and four second segments under different server settings for tile creation, tested on the Meta Quest 2 with24FPS. ................................... 53 1 Introduction Nowadays digital videos are present in all daily lives, present in entertainment, communication, education, and many other fields. As video content consumption continues to grow across different kind of devices and platforms, the efficient transmission of video data is becoming a more important topic. This process includes multiple complex technologies, starting from the capture of the raw video signal, followed by compression into a usable format that retains all important visual information while still reducing the amount of data, which can then be transmitted to certain receiving devices over networks with different bandwidth constraints. Finally, a receiving device needs to be capable of decompressing the data to playback the obtained video on the devices display in such a way that the user gets a visually pleasing experience. Therefore, each stage of this pipeline needs to be optimized to ensure high quality videos without delays, stutter, or visual artifacts. To address the diversity of video applications, a wide range of video compression and streaming techniques were developed. For example, live video streaming requires low-latency transmission and real-time encoding, while with Video On Demand (VOD) compression efficiency and buffering strategies are of much higher importance. Another example would be video conferences, where it is important that there is a consistent frame rate and low bitrates, as well as no stutter and delay, to have interactive communication. Each of these use cases has their own requirements and trade-offs, which are addressed by multiple different techniques such as encoding strategies, network protocols, quality control mechanisms, and many more. In addition, a growing topic in video streaming is the personalized view of the captured and provided video for each individual user. In this case there are mostly panoramic videos where the user can decide which region is of interest, using a phone, a PC or a Virtual-Reality (VR) headset. This interactive and user-driven viewing experience brings completely new approaches to optimize the video transmission pipeline to users. Since they typically only focus on a limited portion of the panoramic scene at any given time, it is inefficient to transmit the entire video at full resolution. By using this knowledge that only a small portion of the whole video will actually be seen, and even a smaller portion 1 2 Chapter 1. Introduction is in the focal area of the human visual field, methods for reducing the video quality in regions which are not directly seen and not in the focal area were introduced. This does not only lead to a reduction of bandwidth requirements for video streaming but also to a reduction of the rendering effort on the clients’ device. This is the point where we want to contribute to ongoing research by proposing an approach for the efficient compression and transmission of panoramic video content. Our method enables personalized viewing by dynamically adjusting the video quality based on the user’s view port, which leads to a reduction of the required data rate without reducing the perceived video quality. Specifically, we try to combine three core aspects which are the optimization of compression techniques which are tailored for panoramic content with individual user region awareness. Then the delivery framework that supports adaptive streaming based on real-time view port tracking to optimize the bandwidth usage, and finally a robust decompression pipeline that ensures a seamless video experience with minimal latency and no recognizable video quality loss. With this approach, our goal is to improve the scalability and user experience, especially for the VR panoramic video use case, which should allow access to panoramic video streams with reduced bandwidth requirements. In the following work, the current State-of-the-Art (SotA) and recent advances in this research field will be reviewed in Chapter 2, as well as the necessary technical background and technologies. The motivation for improving this area, as well as the requirements to achieve our goals, will be provided in Chapter 3. In Chapter 4 our approach to optimizing video transmission will be presented, followed by an evaluation of different settings and the different benefits, as well as a comparison with current SotA and recent research in Chapter 5. Finally, Chapter 6 concludes this thesis and summarizes the results obtained and gives an overview of what can be done to further improve this work in the future. 2 Related Work & Background Contents 2.1 Viewport Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.2 Tile-based Streaming . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.3 BitrateAdaption............................ 5 2.4 Content Delivery Networks . . . . . . . . . . . . . . . . . . . . . 6 2.5 Video Compression Background . . . . . . . . . . . . . . . . . . 7 2.6 Color Space Background . . . . . . . . . . . . . . . . . . . . . . . 11 2.7 Streaming Protocols Background . . . . . . . . . . . . . . . . . . 13 For the delivery of 360°videos, there are solutions in which all 360°content is streamed to users, regardless of the viewport position, which leads to an extreme waste of network resources. To address this problem, viewport prediction, tile-based streaming, and bitrate adaption are important procedures for adaptive 360°video streaming according to Yaqoob et al. [7]. Furthermore, Zhiyu Pang [8] stated that these three topics are currently the main research topics on 360°adaptive streaming. Therefore, this Section will dive into those areas and state the latest relevant findings for this thesis. 2.1 Viewport Prediction The correct prediction of the user viewport is the main priority when it comes to efficient streaming of 360°video content. There exist two main prediction methods which can be classified as motion-based prediction stated by Xie et al. [9] and content-based prediction shown by Fan et al. [10]. Motion-based viewport prediction attempts to extrapolate the future viewport of users with the help of the sensors included in the Head-Mounted Display (HMD) by using historical motion information. According to Qian et al. [11], one problem that occurs here 3 4 Chapter 2. Related Work & Background Figure 2.1: User’s visual fixation heat map on video frames described in Ban et al. [1]. is that the long-term viewport prediction is highly inaccurate and the accuracy decreases along with the prediction period. The content-based method considers the visual saliency of the video to predict the viewport, but this leads to a high computational complexity due to the bias of users Region Of Interest (ROI). Ban et al. [1] came up with an approach in which most people are interested in one object within the picture, and when this is found, there is a good reason why other viewers also tend to have the same ROI. They verified this by analyzing a real head-movement trace dataset which is shown in Figure 2.1 as a heatmap. They combined the benefits of motion-based and content-based prediction by using Linear Regression (LR) for finding a motion-based fixation which then is combined with the content-based approach using the K-Nearest-Neighbors (KNN) algorithm. With that approach, they achieved an improvement of around 30% in bandwidth occupation and viewport quality variance compared to traditional viewport predictions. 2.1.1 Foveated Rendering Another approach to improve video quality and reduce data that need to be handled by an HMD is foveated rendering. According to Wang et al. [12], it takes advantage of the inherent features of human eyes and renders different regions with different qualities without sacrificing perceived visual quality. This concept is based on three main steps. The first step is to apply perceptual models of the human visual system to foveated rendering. The image is then rendered with varying levels of quality based on the foveation principles, and the foveated rendering is integrated into existing rendering paradigms to improve overall performance. This is possible because the fovea region, the region where the visual system perceives the finest details, is just a small region of our whole visual field. As shown by Jabbireddy et al. [2], when using a VR-HMD, we only see that 4% of the screen pixels lie in the foveal region and the rest lie in the peripheral region. The importance in foveated rendering or also in viewport-related rendering can be seen in Figure 2.2, which shows what percentage of pixels are occupied by the fovea region. 2.2. Tile-based Streaming 5 MOST VR PIXELS ARE PERIPHERAL 96% 27% 4% iPhone 7 plus 27" Desktop Monitor 2016 VR HMD Figure 2.2: Percentage of pixel in the foeva region described in Jabbireddy et al. [2]. 2.2 Tile-based Streaming Tile based streaming tries to utilize the known viewport of the user. To do so, the video needs to be separated into multiple parts, in this case into a variable number of tiles, where the client will then receive only the tiles in high quality that are within the ROI and the rest in lower quality, or they can even be neglected. Using the High Efficiency Video Coding (H.265/HEVC) described by Sullivan et al. [13] provides the benefit, when used with tiled videos, that standard video decoders can be used. For streaming video content, the standardization of MPEG Dynamic Adaptive Streaming over HTTP (DASH) described by Iraj Sodagar [14] can be used for 360°video adaption and performs similarly to traditional video content. The 360°video is spatially partitioned into a variable number of non-overlapping rectangular regions, which are the so-called tiles that all hold a Spatial Relation Description (SRD). The SRD provides the spatial location of each video on a global grid, as described by Niamut et al. [15]. This tiled video is then further split into temporal fixed-duration segments. To be able to take advantage of those tiled segments, each tile is encoded in multiple bitrates, so that also multiple clients with different tile requirements can watch the same stream without a needed change on the server side. On the client side, the viewport prediction algorithm decides which tiles to fetch and in what quality. In Figure 2.3, the whole process from getting an input image, encoding and DASH creation, and communication between the client and the server is shown in a simplified way. 2.3 Bitrate Adaption For the client to be able to decide which tiles will be requested from the server, a bitrate adaption logic needs to be used because there is always a upper limit of bandwidth which 6 Chapter 2. Related Work & Background Figure 2.3: Overview of MPEG-DASH streaming with tiling proposed by Concolato et al. [3]. should not be exceeded. In the DASH client implementation of the GPAC Open Source project by Le Feuvre et al. [16] the bitrate adaption logic is divided into three main steps. The first step is tile priority setup, where the DASH client first identifies the coding dependencies of each tile set when loading a Media Presentation Description (MPD) with SRD information. Then the media renderer is informed about the SRD details for each tile to adjust the layout based on the display size. Once the tile sets are identified, the priorities are assigned to the tiles, which is done every time when rate adaption needs to be performed. The second step is the rate allocation, which gathers all the information on the throughput of the network and the bitrates of the tiles. With this information, the available bandwidth is distributed between the different tiles and their priority levels. The big difference to non-tiled scenarios is that the client needs to download multiple video segments at once and therefore decide with which bitrate the tiles are requested. Once the bitrate for each tile is selected, the third step, the rate adaption is performed for each tile with a target bandwidth equal to the selected bitrate. 2.4 Content Delivery Networks Content Delivery Networks (CDNs) have evolved significantly since the beginning of 2000 to provide a fast, reliable, and scalable distribution of digital content. A typical CDN consists of a network of geographically distributed cache servers which are designed to 2.5. Video Compression Background 7 Figure 2.4: Workflow of CDN proposed by Yang et al. [4]. reduce latency by delivering content to the user from the closest available server. When the user requests content, the request is routed through Domain Name System (DNS) and load balancers to select the optimal caching server. When the content is not already cached there, it is fetched from the origin server to deliver it, as explained by Yang et al. [4]. This core concept can be seen in Figure 2.4. As there are multiple different use cases for media content delivery, there are also different types of CDNs, which was shown by Ali et al. [17]. To name some common ones, there are traditional CDNs which rely on dedicated edge servers, global replication, and DNS routing for latency control, such as Akamai Technologies [18], Cloudflare [19] or Amazon CloudFront [20]. Another well known CDN type is the cloud-based CDN which offers more flexible and scalable distribution tightly integrated with cloud services and elastic deployment as, for example, AWS [21] or Azure [22], and lastly there are local CDNs which are mainly used for network-specific optimizations or prototyping with the lack of global reach, such as Nginx [23] or Apache [24]. Of course, this is not the full list of CDNs available out there as Pathan et al. [25] already showed in 2007. 2.5 Video Compression Background Citing Iain Richardson [26], ”Compression is the act or process of compacting data into a smaller number of bits. Video compression (video coding) is the process of converting digital video into a format suitable for transmission or storage, whilst typically reducing the number of bits.” 14 Chapter 2. Related Work & Background delivery channels such as User Datagram Protocol (UDP), multicast UDP, and Transmission Control Protocol (TCP) and is capable of controlling multiple sessions at once. The syntax and structure of the protocol are similar to the HTTP protocol, but add new request methods such as PLAY, PAUSE, SETUP, DESCRIBE, and TEARDOWN. The URL of RTSP is also very similar, with the only difference in the scheme used rtsp:// in RTSP instead of http:// in the HTTP protocol, as described by Santos-Gonz´alez et al. [54]. 2.7.2 Real Time Messaging Protocol The Real Time Messaging Protocol (RTMP) developed by Adobe [55] was designed to provide high-performance transmission of audio, video, and data between the Adobe Flash platform technologies. RTMP is available as an open specification to create products and technologies that enable the delivery of video, audio and data in open AMF, SWF, FLV, and F4V formats which are compatible with Adobe Flash Player. In comparison to the RTSP, which usually works over UDP, RTMP operate on top of TCP, which maintains persistent connection and makes it reliable to transmit all video content. Although Flash technology is now obsolete, RTMP has still a key role as a streaming protocol used, for example, by YouTube Live [56], Twitch [57], or Facebook Live [58]. It is also often used for broadcasting to streaming servers, where it is transcoded into HTTP-based formats for distribution to end users. 2.7.3 HTTP-Livestreaming The HTTP Live Streaming (HLS) protocol developed by Apple Inc. [59] is an HTTP-based adaptive bitrate protocol which is widely used in media players, web browsers, mobile devices, and streaming media servers. HLS splits the overall stream into a sequence of small HTTP-based file downloads, where the stream can be encoded at different bitrates. These files can then be requested by the client using an extended M3U playlist file, which allows the client to decide on the bitrate that the current stream sequence should be loaded according to the current network condition, without causing delays. Due to the use of HTTP as the base, it benefits from existing CDNs and is compatible with most firewalls and NAT setups. Commonly used segment times of 6-10 seconds result in high end-to-end latency, making it less suitable for interactive applications. 2.7.4 Dynamic Adaptive Streaming Over HTTP Not different from the HLS protocol, the DASH protocol developed by MPEG [60] also provides segmented video content at multiple bitrate levels, allowing for adaption to the network condition. This information is provided to the client with a DASH Media Presentation Description (MPD). It is a pull-based bitrate streaming standard in which the 2.7. Streaming Protocols Background 15 client device plays the central role by deciding on the video adaptation. The client can select the HTTP-based file with the highest bitrate possible that can be downloaded in time for playback without causing delays or rebuffering events. Compared to HLS, DASH is codec-agnostic and is standardized as an open international protocol. By providing fine-grained control over segment size and encoding ladders, it is highly suitable for large-scale VOD deployments, which can be seen by the adoption on platforms such as Netflix [61] or Amazon Prime Video [62]. 2.7.5 Low Latency Streaming For both streaming versions, HLS and DASH, low latency adaptions were created with slightly different approaches. The Low Latency DASH (LL-DASH) [63] extension was first introduced in 2018 and was added to the standard in 2019, followed by Low Latency HLS (LL-HLS) [64] introduced in 2019. The main difference between the traditional streaming methods and the low latency ones is the reduction of segment durations, optimizing the segmentation mechanism and the HTTP protocol. Where LL-DASH uses chunked transfer encoding (CTE) to progressively send segments as smaller chunks, LL-HLS uses partial segments, which are smaller video fragments within a full segment that can be requested separately. According to Bentaleb et al. [65], who compared these two different approaches, LLDASH can achieve latencies as low as 1.5-3 seconds with the use of HTTP/1.1 while LL-HLS was able to achieve nearly similar results by the use of HTTP/2. The main problem that came up with the low latency extensions is that the existing adaptive bitrate decisions cannot be used directly, and new solutions for that need to be found. 3 Motivation & Requirements Contents 3.1 Requirements.............................. 18 3.2 Server-Side Requirements . . . . . . . . . . . . . . . . . . . . . . 18 3.3 Client-Side Requirements . . . . . . . . . . . . . . . . . . . . . . 19 The distribution of videos to multiple devices at once, where each user is focusing on a different part of the stream, is gaining importance with the increase of high quality VR devices such as the Apple Vision Pro1or the Meta Quest 32and also on mobile and flat screen devices with Pan-Tilt-Zoom (PTZ) operations to move the virtual camera. This has been developed in both research and industry, which can be seen in the number of papers represented by Jeppsson et al. [66]. Finding the sweet spot between latency, video quality, and bandwidth usage is crucial and very different compared to standard 2D video streaming. One of the main advantages of the H.265/HEVC codec is that the tiling is already in their standard, and when this is combined with DASH-SRD the basis is already there to provide different view qualities to multiple devices. Shown in the work of S´anchez De La Fuente et al. [67], with the use of the H.265 extension scalable High Efficiency Video Coding (SHVC) and multi-view High Efficiency Video Coding (MV-HEVC) the peak streaming bitrate can be reduced while the ROI changes. This is vital for a high immersion experience because stutter or poor quality while moving within a VR device can not only worsen the experience but also lead to cybersickness, which causes symptoms such as nausea, fatigue, headache, stress or vomiting, as described by Martirosov et al. [68]. This led to the idea to create a server application that is capable of tiling H.265/HEVC 360°live-videos received over RTMP and provide it to the clients via DASH with high fidelity. The client application should run on Android devices, including mobile phones and VR headsets, which can receive this stream via the DASH protocol, with only the tiles 1https://www.apple.com/apple-vision-pro/ 2https://www.meta.com/quest/quest-3/ 17 18 Chapter 3. Motivation & Requirements downloaded in high quality that are actually in view, then recreate a fully not tiled video and provide it to the users with nearly real time properties and without any noticeable quality loss due to the lower quality tiles which are not directly seen. 3.1 Requirements Basically the requirements for this project can be separated into two main parts, serverbased and client-based. In this Section a short overview of those main requirements will be shown and later in Section 4, the realization of them will be presented. 3.2 Server-Side Requirements In general the server needs to be able to let an external device stream a live 360°video to it, then process the received video, so it is represented in tiles, each with multiple quality levels, and provide it in an efficient way to the clients, so that they can decide on their own what tile is downloaded in what quality. 3.2.1 RTMP Listener To start with the project, a Linux based server which is capable of creating a stable RTMP listener, which is accessible over the network, so that a camera can send the video data to it, needs to be created. The need for RTMP comes from the fact that it maintains a persistent connection based on top of TCP, which makes it more reliable as described in Section 2.7. 3.2.2 Tiling The Video When the server is capable of receiving the video stream, it now has to process the video into multiple individual tiles and qualities. To be able to do this, a H.265/HEVC encoded video coming from the RTMP listener is required, as the H.265/HEVC standard already makes use of the tiling concept. Further constraints are that the tiling can only be done with a width and height of multiple of 64 and the input format has to be of yuv420p. In addition, the desired number of quality levels and the quality level itself need to be decided in this step. 3.2.3 DASH Creation After having the video stream in multiple quality tiles, they need to be stored on the server in a way that makes them distinguishable so that a client can later easily find the correct tile with regard to the correct time, location and quality level. To make them storable, segments of the full tiled streams need to be saved, as the client needs to be able 3.3. Client-Side Requirements 19 to download them separately. This segment time needs to be declared in this step and with that information the MPD file can be created which allows the client to find all files. 3.2.4 Content Delivery Lastly on the server side, the created files need to be made available over the network so they can really be accessed. For that a file-based web-server needs to be set up which can make the video segments available without much delay. For that, it is important that the read and write operations on the hard drive be reduced to a minimum as they are very costly. Also, the server needs to let the same client download multiple files for the time the live stream lasts, as all tiles of one video segment need to be loaded at every segment interval. 3.3 Client-Side Requirements The client needs to be able to represent the tiled live video to the user with the lowest possible download rate, while still providing the best user experience. To achieve that, an efficient way needs to be found to calculate the correct tile quality and make them visible on the client’s screen in a real-time manner. 3.3.1 DASH Download To start on the client side, the information on how the stream is set up is needed from the server. To get this information, the MPD file is downloaded from the server and the information about video size, tile setup, the segment lengths and video times needs to be extracted. In addition, a linkage between the tile position, tile quality, and the index of the associated files that need to be downloaded needs to be made. 3.3.2 Tile Quality Selection After extraction of the information embedded in the MPD file, the tile configuration can be defined on the client sphere, which is necessary to find out which tiles to download in what quality. Therefore, the field of view of the client needs to be known, as well as the current view direction. With this information, the tile quality of each individual tile can be selected and connected to the file that needs to be downloaded. 3.3.3 Rearranging Downloaded Tiles When the tiles are selected, they need to be downloaded as fast as possible, optimally in parallel. Then, all tiles need to be concatenated with a so-called initialization file in the correct order, which is line-by-line from top to bottom. This concatenated file is then used to recreate a playable video file, which shows all the tiles in the correct order and quality and has a length of the server-defined segment length. 20 Chapter 3. Motivation & Requirements 3.3.4 Displaying The Video After a playable video is created, now the video must be displayed on the client’s device. Therefore, it is necessary that the device is capable of decoding the H.265/HEVC encoded video and show it on the screen without any noticeable delay. As we are talking here mainly about spherical videos for the 360°case, the mapping of the pixel onto the video sphere also needs to be done correctly, so no unwanted tilts or distortions occur on the device. 3.3.5 Seamless Video Playback To achieve a seamless transition between the individual segments of the video, the next video segment must be downloaded, concatenated, and prepared to be displayed on the device before the current segment finished playing. The timing of those procedures needs to be set in such a way that there is no big latency between the currently last available video segment of the video, so we are as close as possible to real-time streaming. At the same time, it must be ensured that the segments are not downloaded too early as the clients view direction can always change and the tile qualities are selected according to it. 4 Implementation Contents 4.1 ConceptualDesign........................... 21 4.2 Server-Side Implementation . . . . . . . . . . . . . . . . . . . . . 23 4.3 Client-Side Implementation . . . . . . . . . . . . . . . . . . . . . 28 4.4 Limitations ............................... 39 To improve the State-of-the-Art H.265/HEVC live streaming, we decided to build a server application which is capable of receiving a live RTMP stream which is then transcoded to a tiled DASH stream of multiple bitrate that can be accessed through a Web server. For the client side, we decided to create a Unity application, which is responsible for the bitrate and tile adaption of the DASH stream, to represent the 360°video in the most efficient way. 4.1 Conceptual Design We started by creating a Linux-based server, which runs the GPAC multimedia framework to be able to run an RTMP server, which is capable of receiving a H.265/HEVC stream from a network camera to fulfill our first server requirement stated in Section 3.2. When the stream is received, we make use of FFmpeg and the Kvazaar encoder to be able to tile the stream into multiple qualities and tile sizes, with what we were able to satisfy the second server requirement. The GPAC framework is then used to generate the DASH files for our tiled stream. This includes the MPD file, which provides the metadata for the stream, the initialization files, which are required for video and audio recreation, the tiled video segments, and the audio files. With that done, only the last requirement on the server side is missing, which is the delivery of the created data to the clients, which we have realized with an Nginx server that is capable of distributing the data on the Web. 21 22 Chapter 4. Implementation Web Server providing Files Multi Quality Tiling HTTP requests/ responses 360° Camera Video Generation Tile selection Position estimation Received tiles Final View RTMP DASH Segments & MPD Generation Sphere mapping Server Client Figure 4.1: Representation of the workflow for live 360°tiled DASH video streaming. The separation of the client and server implementation is highlighted to see what implementation part belongs only to the client or only to the server. Now to start with the client side, we decided to work on an Unity project for Android devices, which can be run on mobile phone devices as well as on HMD-VR devices. To download the correct files from the server, we implemented a MPD parser in C#, which extracts all the information needed to get the correct segment numbers, as well as the correct tile indices for quality control, fulfilling our first requirement as described in Section 3.3. For loading the correct tile, we now need the viewport of the client as well as the tiles which are currently fully visible, partially visible, and non-visible. To get that information, we project a grid with the same tiling as in the MPD file onto a sphere and check via the camera transform and rotation parameters which tiles need to be downloaded in what quality levels. Now, as the second requirement is met, we download the correct tiles and need to rearrange them in a way that we get a playable video, which is done by concatenating the initialization file with the tiles in left to right and top to bottom order. This is then fed into a cross-compiled C++ GPAC library with Kvazaar included, which creates a playable video file out of it. By fulfilling the third requirement, we now have to display the created video to the user, which is done by mapping the video into the inside of a sphere, while placing the camera in the middle of it. With a simple shader, the distortions of the spherical video are corrected so that the user can see the video in good quality. This entire pipeline is visualized in Figure 4.1, where all individual parts can be easily identified. 4.2. Server-Side Implementation 23 As all the individual parts are now done, to finish the application, we needed to find a way to have a seamless video playback, which we did by downloading and preparing the following segment while the current video is displayed to the user. The preparation includes all the previously described tasks, including position estimation for tile selection, downloading and concatenating the correct tiles, combining them to a playable video, and mapping them onto the sphere. Then we wait until the current video is finished and start the whole process again. With that done, all the requirements of our projects described in Chapter 3 are met, and the application is ready to be used. In the following Sections, a detailed version of the implementation will be presented. 4.2 Server-Side Implementation To be able to receive a live 360°video and process it in a way that it can be accessed at multiple bit rates by a client device, we created a server infrastructure based on the Ubuntu 22.04 [69] operating system. The main applications which were needed to achieve this were the GPAC multimedia framework, already briefly introduced in Section 2.3, the Kvazaar encoder developed by Ultravideo [70] and Nginx created by Igor Sysoev [23] which servers as the Web server to distribute the processed data. 4.2.1 Server Setup To start with the setup of the mentioned libraries, the main underlying library FFmpeg introduced by Fabrice Bellard [71] needed to be modified. It is the leading multimedia framework which is capable of decoding, encoding, transcoding, muxing, demuxing, streaming, filtering, and playing pretty much anything video-related. It also serves as the backbone of the used GPAC framework used in this work. For FFmpeg to be able to tile the H.265/HEVC incoming video stream, the Kvazaar library was needed to be compiled into the FFmpeg build together with the x265 extension, which is not enabled by default due to licensing problems which are already stated in Section 2.5.2. With those libraries compiled and ready, the GPAC framework was configured to start an RTMP server via the underlying FFmpeg library. This server waits for a live video, which can then be further processed. When a video is received by the RTMP server, it is piped to the encoders, for the creation of the video tiles to the Kvazaar encoder and for the audio encoding to the Advanced Audio Coding (aac) encoder created by MPEG [72], which is also provided by the FFmpeg library. The output of the encoders is then processed by GPACs DASH filter, which can take several inputs. In our case, multiple tiled H.265/HEVC encoded video streams at different bit rates and one aac audio stream. These are then packed as a live DASH stream containing the MPD file and all needed segment files which are then stored in a dedicated storage space mounted as Temporary File System (tmpfs). 30 Chapter 4. Implementation HTTP requests/ responses Position estimation Tile selection Received tiles Figure 4.4: Tile selection process on the Client-side. the quality selection per tile, where in the optimal case, only the fully visible tiles are downloaded in the best quality, and the further away the tiles are, the lower the quality should get, so there is no sudden drop quality wise. This would of course require the server side to also provide four different quality levels. If there are less quality levels provided by the server, the tiles near the view port get the best quality. Then only the correct segment number needs to be calculated, which is done with the timing information from the MPD file extracted previously. With that information, it is now possible to download all individual tiles and the appropriate audio segment to create the video segment. 4.3.3 Creation Of A Playable Video Now the identified files are downloaded, where each download runs in a separate co-routine to be able to download them in parallel. When all files are there, the init file, which we already have from the parsing process, is concatenated with the tiles in left-to-right, topto-bottom order. Separately, the audio file is also concatenated with its init file, which then gives us two concatenated files in total. To get a playable video and audio format, these two files need to undergo the reversed process which is done on the server. For that we cross-compiled the GPAC library with the Kvazaar encoder embedded for Android arm64 devices. This was done with openjdk-8 and Androids Software Development Kit (SDK) 32 and Native Development Kit (NDK) 24 to be compatible with the current Android version running on the Meta Quest devices. With this cross-compiled library, we create a native C++ Dynamic Link Library (DLL) which can take the concatenated video file and produce a playable file in the mp4 format and also the concatenated audio file that produces a playable audio file in the wav format. This was done by creating an input filter that processes the concatenated file. The filter is then connected to the tileagg filter, which can take Kvazaar created tiles and combines them into a playable video, and finally a connection to a destination filter for the output. The way this looks like for the aggregation of the video file is shown in the Listing 4.5. In Unity, we have to create the folder structure Assets/Plugins/Android/ where the created .so file should be placed, as well as all library dependencies. To access this native library from Unity, the specific function must be imported with a DllImport, which is 4.3. Client-Side Implementation 31 1extern "C" __attribute__((visibility("default"))) 2int runTileAggregator(const char*inputFilePath, 3const char*outputFilePath) 4{ 5GF_Err gf_err =GF_OK; 6GF_FilterSession*session=gf_fs_new_defaults( 7(GF_FilterSessionFlags)( 8GF_FS_FLAG_NO_REGULATION | 9GF_FS_FLAG_NO_PROBE 10 )); 11 12 GF_Filter*src_filter =gf_fs_load_source(session, 13 inputFilePath, NULL,NULL,&gf_err); 14 GF_Filter*tileagg_filter =gf_fs_load_filter(session, 15 "tileagg",&gf_err); 16 GF_Filter*dst_filter =gf_fs_load_destination(session, 17 outputFilePath, NULL,NULL,&gf_err); 18 19 gf_filter_set_source(tileagg_filter, src_filter, NULL); 20 gf_filter_set_source(dst_filter, tileagg_filter, NULL); 21 22 gf_err =gf_fs_run(session); 23 gf_fs_del(session); 24 25 return gf_err; 26 } Listing 4.5: Reduced version of the C++ Dll for Android, to reassemble the concatenated tiles into a playable video. The error handling is removed from this listing for better visibility. shown in the Listing 4.6. After calling the aggregation function for the video and audio file, we are ready to show the result on the client device. 4.3.4 Displaying The Video As all the needed files are now present on the client device, the video player must be prepared to display the video file in mp4 format as well as the audio clip must be loaded. To do so, a video player game object is created in our scene which is attached to the previously mentioned sphere. To make sure that the video is displayed correctly later on, a shader was created which maps the individual pixels to the inside of the sphere. Then the prepare function is called, and we wait for it to return. For audio, an audio source is added to the camera game object, and the created audio file in wav format is set. When 32 Chapter 4. Implementation 1public class GPACWrapper : MonoBehaviour 2{ 3[DllImport("gpac_tileagg")] 4private static extern int runTileAggregator( 5string videoInputPath, string outputPath); 6... 7private IEnumerator GenerateNextVideo(int index) 8{ 9string catVideoPath =catVideoPaths[index]; 10 string vidPath =videoPaths[index]; 11 Task.Run(() => 12 { 13 int result =runTileAggregator(catVideoPath, vidPath); 14 }); 15 } 16 } Listing 4.6: Reduced version of the Dll import in Unity and call of the native function to recreate a playable video out of the concatenated file. the prepare function is finished, the application is ready to start the playback of both files so that the client can see and hear what is captured from the camera. Now, all individual parts are finished as illustrated in Figure 4.5. 4.3.5 Stage 1: Creating A Seamless Playback Experience Now as all individual parts are there, seamless video playback must be ensured. In the individual steps explained above, all of them run sequentially, which can of course not create a seamless experience, as the whole download and preparation process takes its time. So to pass this problem, a second video player, as well as a second sphere, were introduced in the scene to be able to alternate between them. With them set up, when starting the application, the MPD file is first downloaded and parsed. Followed by the first video player that starts to calculate the visible tiles, downloads and concatenates them, and prepares the video player until it is ready to play. When starting the playback, the second video player starts with its process, again downloading and preparing the following segments, but when the preparation is finished, it does not automatically start the playback, it waits until the already running player is finished. After the first video player has finished its segment, the associated sphere is disabled, and the sphere of the second video player is activated, the new audio file is loaded, and the playback of the second player starts. The first player then starts the process of retrieving and preparing again. With this procedure, it is possible to create a seamless transition between the individual DASH segments produced by the server, resulting in a delay that is approximately three times as large as the segment length defined by the server. 4.3. Client-Side Implementation 33 Web Server providing Files Multi Quality Tiling HTTP requests/ responses 360° Camera Video Generation Tile selection Position estimation Received tiles Final View RTMP DASH Segments & MPD Generation Sphere mapping Individual parts managed by the Client-side implementation Figure 4.5: Overview of the individual parts managed by the Client-side implementation. Reducing the segment length enabled near real-time video transmission, but also revealed limitations of the Unity video player. The preparation phase required up to three seconds, leading to a minimum segment time of around four seconds, as downloading and reassembling the files required additional 300 milliseconds. This constraint not only led to delays of roughly ten seconds, but also caused noticeable degradation when changing the viewport, as a high quality image at the current viewport is only created after the completion of the current and following segment. 4.3.6 Stage 2: Improve Seamless Playback With Native Video Players The initial setup enabled a working prototype, but to get closer to real-time streaming, further optimizations were required. Therefore, we decided to create our own video player as there is no way to tweak the performance of the Unity built-in player. For that we used the Android Media library, which gives the possibility to directly decode the video frame by frame on hardware encoders, as the current generation has H.265/HEVC included. We created another C++ DLL which is called from a C# bridge, responsible for the communication between Unity and the DLL. The video player gets initialized with a texture created by Unity, which is attached to the sphere, together with the size parameters and the current video path. Then the Android Media library opens the video file and loads it into one of the hardware decoders to decode the first frame. This produces an image in the YUV format as the output, which 34 Chapter 4. Implementation cannot be directly displayed by Unity’s texture. To address this problem, the YUV image needs to be converted to the RGBA format that can be displayed, resulting in extensive calculations on the CPU. When the first frame is converted,a message is sent back to Unity that the preparation is finished. This allows the call to perform the texture update, which needs to be triggered by Unitys render thread, to write the data onto the texture. With that version, it is possible to achieve a preparation time of less than one second, as loading the first frame is still a time-consuming task, but so we are able to reduce the segment time by half compared to our first approach. However, the conversion from YUV to RGBA remained a bottleneck. Even with a parallelized optimize approach, the process required 25 to 35 milliseconds per frame, which is enough to display a 24 Frames Per Second (FPS) video but inadequate for 30 FPS, where delays accumulate over time. To address this limitation, a frame drop mechanism was integrated into the decoding loop, which ensures that only frames that could be processed in time were displayed, while late frames were discarded. This setup enabled a seamless stream with minimal delay and without noticeable frame drops during segment switches. By reducing the segment length to two seconds, the system achieved a total latency of approximately six to seven seconds between the camera capture and the video displayed at the client. To show the communication of the individual players in the implementation, Figure 4.6 illustrates the individual parts of the program with the time consumed at each step. It also represents the way in which the two players communicate with each other to have a seamless switch between them. 4.3.7 Stage 3: Optimize Playback Pipeline With SIMD To maintain a stable framerate of 30 FPS on devices without dropping frames while processing the video, standard parallelization and loop unrolling proved insufficient. Therefore, arm neon instructions were used to convert YUV to RGBA using the Single Instruction, Multiple Data (SIMD) technique on devices that support it. This technique enabled the ability to process eight pixels at a time, allowing 4K video frames to be converted within 15 to 20 milliseconds and therefore avoiding frame drops. Since the conversion speed is faster than the required display rate, an additional synchronization step was introduced. Instead of dropping frames, as in the earlier approach, the system now delays the output until the appropriate display time. In Figure 4.7, the process from the initialization of the application on the client, until ten seconds of video processed and displayed on the device, can be seen. Every function call on the client is listed, as well as the frames produced and displayed, which end up being 300 frames after ten seconds, showcasing that not a single frame is dropped. What is interesting to see here is that the time consumption for the frame generation is still on the upper limit while the second video decoder is preparing. This is visualized by the color coding in the Framecount line, where green indicates that the frame was generated fast enough, and red took too long. As none of them took twice the time and 4.3. Client-Side Implementation 35 Program Start Dash Parser Start Downloaded MPD 63 ms Parsed MPD 378 ms Downloaded Init Files 24 ms Player 1 Start Calculated Visible Tiles 2 ms Downloaded Tiles 142 ms Concatenated Tiles 4 ms Downloaded Audio 22 ms Concatenated Audio 4 ms Tiles Aggregated Audio Aggregated Decoder Initialized Video Track Found Video Info Extracted Input Buffer Filled Output Buffer Filled YUV to RGBA Converted Preparation finished Start Video Video Finished Player 2 Start Calculated Visible Tiles 2 ms Downloaded Tiles 168 ms Concatenated Tiles 6 ms Downloaded Audio 25 ms Concatenated Audio 2 ms Start Video Video Finished 73 ms 54 ms 8 ms 30 ms 27 ms 199 ms 680 ms 23 ms 12 ms Tiles Aggregated Audio Aggregated Decoder Initialized Video Track Found Video Info Extracted Input Buffer Filled Output Buffer Filled YUV to RGBA Converted Preparation finished 72 ms 50 ms 12 ms 43 ms 26 ms 214 ms 570 ms 20 ms 8 ms Program Start Dash Parser Start Downloaded and Parsed MPD and Downloaded Init Files 465 ms Player 1 Start Calculated Visible Tiles 2 ms Downloaded and Concatenated Tiles + Audio 172 ms Tiles and Audio Aggregated Decoder Initialized and First Frame loaded Start Video Video Finished Start Video Video Finished 127 ms 979 ms Player 2 Start Calculated Visible Tiles 2 ms Downloaded and Concatenated Tiles + Audio 201 ms Tiles and Audio Aggregated Decoder Initialized and First Frame loaded 122 ms 893 ms Program Start Downloaded MPD 63 ms Parsed MPD 378 ms Downloaded Init Files 24 ms Player 1 Acquire Data Calculated Visible Tiles Downloaded and Concatenated Tiles + Audio Tiles and Audio Aggregated 2 ms 172 ms 127 ms Player 1 Prepare Video Load and Extract Video Decoder Input and Output Buffer filled Convert First YUV Frame to RGBA and Load to Texture 65 ms 879 ms 35 ms Player 1 Play Video Decode Frame Convert YUV to RGBA and Load to Texture Player 2 Acquire Data Player 2 Prepare Video Calculated Visible Tiles Downloaded and Concatenated Tiles + Audio Tiles and Audio Aggregated 2 ms 201 ms 122 ms Load and Extract Video Decoder Input and Output Buffer filled Convert First YUV Frame to RGBA and Load to Texture 81 ms 784 ms 28 ms Player 1 Video Finished Player 2 Play Video Decode frame Convert YUV to RGBA and Load to Texture Player 2 Video Finished Program Start Downloaded MPD 63 ms Parsed MPD 378 ms Downloaded Init Files 24 ms Player 1 Acquire Data Calculated Visible Tiles Downloaded and Concatenated Tiles + Audio Tiles and Audio Aggregated 2 ms 172 ms 127 ms Player 1 Prepare Video Load and Extract Video Decoder Input and Output Buffer filled Convert First YUV Frame to RGBA and Load to Texture 65 ms 879 ms 35 ms Player 1 Play Video Decode Frame Convert YUV to RGBA and Load to Texture Player 2 Acquire Data Player 2 Prepare Video Calculated Visible Tiles Downloaded and Concatenated Tiles + Audio Tiles and Audio Aggregated 2 ms 201 ms 122 ms Load and Extract Video Decoder Input and Output Buffer filled Convert First YUV Frame to RGBA and Load to Texture 81 ms 784 ms 28 ms Player 1 Video Finished Player 2 Play Video Decode frame Convert YUV to RGBA and Load to Texture Player 2 Video Finished Figure 4.6: Representation of the client side workflow, including Unity and native libraries with timings from two video segments. not many of them took to long after another, no frame was needed to be dropped. In addition, a detailed view of the start of the first video segment is shown to visualize frame processing and what happens in the meantime on the second video player. What also can be seen here is that the main bottleneck is still the decoding of the first frame, which is part of the preparation process, as this limits us to reduce the segment time down to one second. Also, here the frame counter is shown in the individual parts, to showcase that 60 frames are processed and displayed within the two seconds of the first video. With this implementation, a stable 4K 360°video stream, captured at 30 FPS from a camera, can be processed and displayed. It is tiled and converted into the DASH format on a server and downloaded, processed, and displayed on a client device with around six seconds of latency when using two-second segments. 36 Chapter 4. Implementation 0123456789101112 Time (s) Download MPD Parsed MPD Downloaded init files Calculated visible tiles Downloaded all tiles Concatenated all tiles Downloaded audio Created tiled mp4 Created audio wav Created decoder Loaded video in decoder Decoded first frame Converted first frame Start video playback Framecount Function time consumption Quelle Player A Player B (a) Ten second diagram of video stream showing the main function calls and their time consumption. Frame generation time is colored in red if it exceeded the expected frame time. 1.6 1.8 2.0 2.2 2.4 2.6 2.8 3.0 3.2 3.4 3.6 3.8 4.0 Time (s) Calculated visible tiles Downloaded all tiles Concatenated all tiles Downloaded audio Created tiled mp4 Created audio wav Created decoder Loaded video in decoder Decoded first frame Converted first frame Start video playback Framecount 61 62636465 6667686970 71 72 73 7412 3 4 5 6 7 8 9 10 11 121314 15 16 1718192021 222324252627 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 4445 46 47 48 49 50 51525354 5556575859 60 Function time consumption Quelle Player A Player B (b) Detailed view of frame generation and function calls. Frame generation time is colored in red if it exceeded the expected frame time. Figure 4.7: Timing diagram of function calls with SIMD implementation and CPU YUV to RGBA conversion tested on Meta Quest 3. 4.3. Client-Side Implementation 37 1void NV12ToRGBABuffer(uint8_t *yuv, uint8_t *rgba, int width, int height) 2{ 3int frameSize =width *height; 4for (int j= 0;j<height; j++) 5{ 6uint8_t *rgba_ptr =rgba +j*width * 4; 7uint8_t *y_ptr =yuv +j*width; 8uint8_t *uv_ptr =yuv +frameSize +(j >> 1)*width; 9for (int i= 0;i<width; i += 2) 10 { 11 uint8_t u= *uv_ptr++; 12 uint8_t v= *uv_ptr++; 13 *rgba_ptr++ = *y_ptr++; 14 *rgba_ptr++=u; 15 *rgba_ptr++=v; 16 *rgba_ptr++ = 255; 17 *rgba_ptr++ = *y_ptr++; 18 *rgba_ptr++=u; 19 *rgba_ptr++=v; 20 *rgba_ptr++ = 255; 21 } 22 } 23 } Listing 4.7: Packing YUV data into an RGBA buffer, to be able to transfer the data to Unitys shader. 4.3.8 Final Stage: Optimizing CPU Load With Shader-Based Color Conversion. The playback of 4K 360°videos at the desired frame rate of 30 FPS was achieved without the need for frame drop, although a small number of frames were occasionally generated too late. However, the last limitation was the inability to reduce the segmentation time to one second. Since color conversion was still performed on the CPU, even with optimized SIMD instructions, this task represented a computational bottleneck that could be achieved with much higher efficiency by the GPU. To overcome this, the YUV data produced by the decoder was packed into an RGBA texture buffer, as Unity on Android devices only allows the creation of RGBA textures. This was accomplished by writing two pixels for every pair of Y values sharing the U and V components, and performing direct pointer assignments instead of the full per-pixel calculations on the native C++ side. The function implementing this mechanism is shown in the Listing 4.7. With this change, even without optimizing it with SIMD instructions, it was possible 38 Chapter 4. Implementation 1float3 YUVtoRGB(float y, float u, float v) 2{ 3float c=298*((y * 255.0)- 16); 4float d=(u * 255.0)- 128.0; 5float e=(v * 255.0)- 128.0; 6 7float r=(c +459*e+ 128)/ 256.0; 8float g=(c -55*d-136*e+ 128)/ 256.0; 9float b=(c +541*d+ 128)/ 256.0; 10 11 return float3(r, g, b) / 255.0; 12 } 13 14 fixed4 frag(v2f i) :SV_Target 15 { 16 float3 norm_dir =normalize(i.direction); 17 18 float2 uv; 19 uv.x =0.5+atan2(norm_dir.x, norm_dir.z) /(2.0 * UNITY_PI); 20 uv.y =0.5-asin(norm_dir.y) /UNITY_PI; 21 22 float4 packed =tex2D(_MainTex, uv); 23 float3 rgb =YUVtoRGB(packed.r, packed.g, packed.b); 24 return float4(rgb, 1.0); 25 } Listing 4.8: Shader implementation, to unpack the YUV data, transmitted over the RGBA buffer, transformed to be displayed inside a sphere and converted to correct values for Unitys RGBA texture. to reduce the CPU processing time from 15 ∼20 milliseconds when doing the complete conversion with SIMD, to 8 milliseconds, since just assigning pointers is much faster than doing the entire calculation on the CPU. To display the packed data, a custom shader was implemented that unpacks the YUV values from the RGBA buffer and computes the final RGB values for each pixel of the texture. This process, including the access of each pixel, the transformation to the inside of the sphere, the loading of the corresponding YUV values from the texture buffer and the conversion of them into RGB values, is presented in the Listing 4.8. With this implementation, the computational effort was shifted from the CPU to the GPU, allwoing all frames to be produced in time, even while the second video player was preparing. The reduced CPU workload also lowered the computation time of all other function calls, as they were performed in parallel. A further optimization introduced a SIMD version of the YUV packaging, which reduced the packaging time to 3 milliseconds, 4.4. Limitations 39 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Time needed (s) Calculated visible tiles Downloaded all files Created wav + mp4 Initialized Decoder Decoded first frame Frame conversion time Comparison of function durations Method CPU Conversion SIMD Pointer Pointer SIMD Figure 4.8: Comparison of function durations tested on the Meta Quest 3. as well as the setup of the decoder and the decode process of the first frame, as illustrated in Figure 4.8. This final client-side implementation enables the playback of a full 360°video recreated of a tiled DASH stream with 30 FPS, by utilizing the CPU for packaging the YUV data and using the GPU for the calculations of the pixel values. This allows a reduction of the segment time to one second, as the process starting from tile selection until the conversion of the first frame can be performed in under one second, which is presented in Figure 4.9. Although the system can process one-second segments in time under controlled conditions, this configuration remains at the performance limit. In practice, it is still preferred to stay with two-second segments to ensure a seamless live streaming experience. Unfortunately, the reduction of CPU resources did not have a big impact on the Android Media library used to decode the first frame, which is still our most time-consuming step in the preparation phase. 4.4 Limitations While this Chapter provides all the necessary steps to create a functional 360°live experience for high quality content delivery, there are also some limitations that must be considered. These limitations arise on both sides, the server implementations, which face problems in handling the data efficiently, as well as the client where limitations with reassembling the downloaded files and efficiently decoding the produced video arise. Also, 46 Chapter 5. Evaluation & Use Cases Tile Layout CPU (%) Memory (%) 4x4 66 3.7 8x2 67 3.7 8x4 67 3.6 16x2 67 3.7 16x4 69 4.0 Table 5.4: The effect of different tile layouts on CPU and Memory usage. Tested with a resolution of 3840×1920, a framerate of 30 FPS, an upload rate of 20Mb/s, a GOP size of 8, Frame Parallelism of 2 and two tile qualities with 20Mb/s and 1Mb/s. Resolution Rate (Mb/s) Layout FPS Qualities (Mb/s) CPU (%) Memory (%) 3840x1920 20 8x4 30 20,1 67 3.6 3840x1920 20 8x2 30 20,1 67 3.7 3840x1920 20 8x4 24 20,1 62 3.7 7680x3840 20 8x4 24 20,1 100 12.5 7680x3840 10 8x2 24 10,1 100 11.4 7680x3840 10 8x4 24 10,1 100 11.5 Table 5.5: The effect of different resolutions on CPU and Memory usage. Tested with a GOP size of 8 and Frame Parallelism of 2 40 Mb/s only has a small influence on the CPU usage compared to adding another tile quality. What is also worth mentioning is that the difference in memory usage is not large in any of the tests and is always below 5%, which shows that this is not a limiting factor. The test where the CPU usage was above 90 % sometimes started to lag behind, as there are also other services running on the server, as well as the one given with 100 % should only show that it was unable to execute in time to generate a live experience on the provided hardware. To evaluate the effect of different tile layouts, we set up our server with a fixed resolution of 3840×1920, a framerate of 30 FPS an upload rate of 20Mb/s and two tile qualities with 20Mb/s and 1Mb/s, where the results are represented in Table 5.4. Interesting to see is also the fact that a tiling setup with a multiple of cores available does not have much influence on the server resources. This allows us to always use a high tiling setup with up to 64 tiles without much difference from using 16 tiles. As the last test to evaluate the server configurations, we used different resolutions as well as different upload rates, framerates, and tiling setups. There only the GOP size and Frame Parallelism remained the same as in the other tests. The results shown in Table 5.5 unfortunately show that our test server was not capable of tiling and creating the DASH format for 8K video in time, as even with the lowest upload rate possible on our camera and only two tile qualities, the CPU was locked at 100% which led to an always increasing delay in the video stream. Still, the memory usage received by htop and shown in the Table can be seen as realistic values, which shows an increase just a little bit smaller than expected, as it has four times the pixels to process. 5.3. Video Quality Comparison 47 5.3 Video Quality Comparison As the download rate is approximately around 20% with the tiled version compared to the standard full 360°DASH streaming, it becomes possible to increase the bit rate on the camera and server side, resulting in four to five times better video quality for clients. Alternatively, the same data rate as non-tiled DASH streaming can be used to connect four to five times more clients to a single server. In Figure 5.2, two versions of a 4K video are shown. In the upper image, the tiled video is created with 50Mb/s upload from the camera and then tiled into 8 ×4 tiles with 50Mb/s and 1Mb/s. This results in 1.57Mb/s high quality tiles for those inside the FoV and directly connected to them and 0.03Mb/s low quality tiles for tiles that are currently not seen. On the client side, this leads to an approximate download rate of 10Mb/s. The lower image shows a version where no tiling is done and the entire 360°video is created with 10Mb/s from the camera and then transferred via the DASH protocol to the client. When comparing these two frames, the lower quality can especially be seen on borders, as they start to create a staircasing effect, which is shown in the zoomed in detail on the left. The overall noise is also much more present, as shown in the detailed view on the right. 5.4 Download Comparison To show the advantage in regards to network traffic, Figure 5.3 shows the download rate for a 30 second video stream while also showing the actual rotation of the view and the amount of tiles which are loaded in full quality. In this case, it is tested with a 20Mb/s camera upload which is tiled into 8 ×4 tiles with 20Mb/s high quality and 1Mb/s low quality. The correlation between high quality tiles needed and the download rate is nicely visualized. It can be seen that the maximum download rate is at 12 Mb/s under heavy movement when downloading 20 tiles in the highest quality, which is mostly an edge case, as when looking at a 360 video, the POI does not suddenly change. This can also be seen in this graph, as it shows that in the average case, the download rate is between 6 and 8 Mb/s with 9 to 12 high quality tiles needed to be downloaded. In Figure 5.3, above and below the graph, two frames of the tiled video are shown, where in the upper one, the HMD was looking at the center of the image and therefore 16 tiles needed to be downloaded in the highest quality, which are marked here with the red outlines resulting in a download rate of 10Mb/s. This frame refers to the start of the graph above. The frame below refers to the lowest amount of tiles shown in the graph, as well as the lowest download rate, where the HMD looked slightly up-left, resulting in only 9 tiles downloaded in high quality outlined in red. As this only shows the situation of one tiling setup, the following Table 5.6 gives an overview of what tile settings are useful and what setup to avoid. The tests were performed with a 4K video send to the server with 20Mb/s upload from the camera. For the client, 48 Chapter 5. Evaluation & Use Cases (a) Tiled video with 50Mb/s upload and 10Mb/s download (b) Full 360 video with 10Mb/s upand download Figure 5.2: Quality comparison of the tiled DASH 360°4K video and non-tiled version with the same download rate per device. we have four different tile visibilities, which are fully visible, partially visible, surrounding, and non-visible tiles that define the quality downloaded as described in Section 4.3.2. As client device, a Samsung A53 with an Android build was used, because it was not possible to get the same camera movement for the test with a HMD. For tracking network traffic, the iftop tool created by Paul Warren [76] in 2002 was used. The Table also shows that a tile layout with only two tiles in one axis is not optimal, as no detailed selection can be done and most of the tiles are counted as partially visible tiles. This does not lead to a big problem when there are only two or three quality levels, but when the server is capable of producing all four quality levels, the highest quality 5.4. Download Comparison 49 12:08:30 12:08:34 12:08:38 12:08:42 12:08:46 12:08:50 12:08:54 12:08:58 0 2 4 6 8 10 12 Download Mb/s Download Rotation X High Quality Tiles 0 50 100 150 200 250 300 350 Rotation ° 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 HQ Tiles Download with camera rotation and high quality tile count Figure 5.3: Download rate of tiled 20Mb/s video with example tiled video. 50 Chapter 5. Evaluation & Use Cases Server Client FPS Layout Qualities (Mb/s) Tile visibility Tile Quality Rate (Mb/s) 30 8x4 20, 1 2, 5, 5, 20 12, 20 7.0 24 8x4 20, 1 2, 5, 5, 20 12, 20 6.1 30 8x2 20, 1 0, 6, 0, 11 6, 10 6.9 24 8x2 20, 1 0, 6, 0, 11 6, 10 6.0 30 16x2 20, 1 0, 10, 0, 22 10, 22 5.3 24 16x2 20, 5, 1 0, 10, 0, 22 10, 22 4.5 30 16x4 20, 1 4, 7, 7, 46 18, 46 4.2 24 16x4 20, 1 4, 7, 7, 46 18, 46 3.5 24 16x4 20, 5, 1 4, 7, 7, 46 11, 7, 46 2.9 Table 5.6: Client download rates with different quality and tile configurations, a fixed resolution of 3840 ×1920 and an upload rate of 20Mb/s. The tile visibility indicates the number of fully visible, partial visible, surrounding and non visible tiles selected by the client, and the tile quality specifies how many tiles are selected at each quality level. tiles would not be loaded. What is also shown there is that the more tiles there are, the better the tile selection can be made, and therefore the data rate can be minimized to its optimum. 5.5 Video Conversion Efficiency One big topic in this thesis was the efficiency of displaying the created video on the client device. As shown in Section 4.3, we approached this in multiple different ways, as there was no straightforward way to implement this on Android devices with the tools provided by the Unity Engine. We created a native video player with the Android Media library, where we can directly communicate with the decoder, which outputs us a YUV image of every frame. This image needs to be converted into the RGBA color space to be displayable on Android devices. In the first version, we converted the color space on the CPU with a parallel approach, which gave us a conversion time of approximately 30 ms on different devices. As shown in Figure 5.4a, which shows the time consumed by each function as well as the frames produced, we can see that we had to drop some frames, indicated in purple, as the Meta Quest 2 was unable to consistently create 30 frames per second in time. The abbreviations of the following figures, which show the function timings for different conversion techniques, are described in Table 5.7. As we also had the Meta Quest 3 as a testing device, which is capable of arm-based SIMD instructions, we tried to optimize the conversion on the CPU for that device, so no frames need to be dropped any more. With that implementation, we were able to reduce the average conversion time to 18 ms, which allowed us to display a 4K video with 30 fps without the need of dropping some frames. However, some frames were generated too 5.5. Video Conversion Efficiency 51 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Time (s) SVP CVT DAF CWM IDE DFF CFF FCO Function time consumption Quelle Player A Player B (a) Function times tested with CPU color conversion. 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Time (s) SVP CVT DAF CWM IDE DFF CFF FCO Function time consumption Quelle Player A Player B (b) Function times tested with CPU packaging and GPU conversion. Figure 5.4: Timing diagram of function calls on a 4K video at 30 fps with different color conversion techniques, tested on Meta Quest 2. The color of the FCO indicates, whether the frame was generated in the correct time shown in green, red, if it was generated too late, and purple, if a frame needed to be dropped. 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Time (s) SVP CVT DAF CWM IDE DFF CFF FCO Function time consumption Quelle Player A Player B (a) Function times tested with CPU color conversion and SIMD instructions. 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Time (s) SVP CVT DAF CWM IDE DFF CFF FCO Function time consumption Quelle Player A Player B (b) Function times tested with CPU packaging with SIMD instructions and GPU conversion. Figure 5.5: Timing diagram of function calls on a 4K video at 30 fps with different color conversion techniques, tested on Meta Quest 3. The color of the FCO indicates, whether the frame was generated in the correct time shown in green, or red, if it was generated too late. Abbreviation Description CVT Time needed to calculate visible tiles and their layout SVP Start of video playback. DAF Time needed to download all files, including tiles and audio. CWM Time needed to create WAV and MP4 files ready to be played. IDE Time needed for the initialization of the Decoder. DFF Time needed for decoding the first frame of the video segment. CFF Time needed for the conversion from YUV to RGBA of the first frame. FCO Frame counter, visualizing the start and end of each displayed frame. Table 5.7: Abbreviation description for timing diagrams of different conversion techniques. late, which means that not all frames had the same display time, as shown in Figure 5.5a. The interesting thing to see here is that the frames, which took too long, only occur, while the second video player was preparing and also needed some CPU resources. Also, 52 Chapter 5. Evaluation & Use Cases when comparing the time needed by the decoder for decoding the first frame as well as downloading the files, between the Meta Quest 2 and 3, it can be seen that the predecessor only needs a fraction of time. Overall, this results in a preparation time of around 0.6 seconds on the Meta Quest 2, compared to more than one second on the Meta Quest 3. Since we were still unable to produce all frames in time, and the task of color conversion can be computed much faster on the GPU, we created a way to transfer the YUV data from the native library to the Unity shader. For this, the data was packed into an RGBA buffer by doing pointer assignments on the CPU, enabling the computational part to be performed on the GPU. With that version, we were able to achieve an average conversion time of 8 ms on the Meta Quest 2, which completely removed the need of dropping frames, and also no frames where generated too late. This also allowed us to reduce the segment time to one second, as the preparation is also always completed in the available time, as seen in Figure 5.4b. In the same way that we implemented the SIMD version for the Meta Quest 3 for the CPU, we also implemented a SIMD version for packaging on the CPU and conversion done by the shader on the GPU, as the frame generation time was significantly reduced in our previous approach. The SIMD implementation with shader side computation reduced the average frame generation time to 3 ms, which can be seen in Figure 5.5b at the conversion time of the first frame. This version also improved our overall computation time for the second video player, when preparing the next video segment to approximately one second, which theoretically enables us to also use a segment length of one second. Still, it is at the upper limit do not create a delay, if one function call takes a bit longer than expected, therefore, to create a seamless playback experience on the Meta Quest 3, two second segments are still recommended. Interestingly, on the Meta Quest 2, were we able to process a one-second segment live stream without a delay caused by the preparation phase, as the decode process of the first frame is still faster there. To provide an overview of the different color conversion techniques and their time consumption for frame conversion, as well as the time needed to decode the first frame, Table 5.8 was created. It includes results from the three test devices, as well as the four versions tested on the Meta Quest 3. Interestingly, the Meta Quest 2 outperforms the other devices no matter what conversion technique is used when looking at the decoder time for the first frame. In addition, even the Meta Quest 3 is only faster than the midrange smartphone when fully utilizing the SIMD functionality combined with our YUV packaging method. 5.6 Video Transmission Latency As this thesis is about real-time streaming, we also have to analyze the delay between recording the video on the camera, until it is displayed on the client device. There are multiple factors which influence this time, the biggest one is definitely the segment time, as 5.6. Video Transmission Latency 53 Testing Device Method Conversion Time (ms) Decoder Time (ms) Samsung A53 Conversion 31 673 Samsung A53 Pointer 12 651 Meta Quest 2 Conversion 27 182 Meta Quest 2 Pointer 8 195 Meta Quest 3 Conversion 25 689 Meta Quest 3 Pointer 8 745 Meta Quest 3 Conversion SIMD 18 791 Meta Quest 3 Pointer SIMD 3 564 Table 5.8: Average frame conversion times and Decoder times for processing the first frame of a segment after 120 frames of different conversion techniques, tested on Meta Quest 3, Meta Quest 2 and Samsung A53. Conversion stands for directly converting the YUV data to RGBA on the CPU, Pointer stands for packaging the YUV data to the RGBA buffer and convertion done on the GPU. Tile Layout Qualities (Mb/s) Delay 1 (s) Delay 2 (s) Delay 4 (s) 8x4 20, 1 5.21 6.85 9.79 8x4 10 4.99 6.72 10.16 8x4 20, 5, 1 5.41 6.98 10.72 16x4 20, 1 5.45 7.07 9.84 4x2 20, 1 5.62 6.68 10.61 Table 5.9: Showing the Latency differences for one, two and four second segments under different server settings for tile creation, tested on the Meta Quest 2 with 24 FPS. the earliest moment the server can provide the latest segment is after the defined segment time. In addition, the camera takes some time to process the image before it can be delivered over RTMP to the server. The server then needs additional time to process the stream into the different tiles and qualities, and finally, the client device needs to download the segment and process the data, to show it to the client. In Table 5.9, multiple different segment settings are shown, to see the difference to the current State-of-the-Art (SotA) version of low latency streaming as described in Section 2.7.5, where they reach latencies of under 3 seconds. As shown in this Table, latencies that low could not be achieved, even when using only one-second segments, which resulted in delays of around five seconds. However, achieving the lowest possible latency was not the purpose of this thesis. It should be noted that different tile layout and quality settings did not significantly affect the delay. No clear pattern indicating which configuration produces the lowest latency was observed in the three different segment lengths. These variations can largely be explained by the start time of the application, as it was not possible to create an environment where the client starts at the exact same time as the newest segment was created. The impact of segment length on latency is more apparent. An increase from one second to two second segments resulted in approximately 1.5 seconds higher latency, and 54 Chapter 5. Evaluation & Use Cases another 3.5 seconds when raising the segment duration to four seconds. This demonstrates that achieving a low segment time not only gives the user a more responsive experience, as the quality adjustment time of the viewport is directly linked to the segment time, but also gives the best real-time experience available with a tiled streaming approach. 5.7 Limitations This Chapter showed what our created setup is capable of, what the most efficient settings are for different use cases, and where our hardware was not able to perform all tasks in the available time. Also, due to limited computational resources and hardware limitations, only a subset of desired scenarios could be evaluated, but what can be said overall is that the CPU usage of Kvazaar for the tiling was the most limiting factor on the server side, and the Android Media library restricted us on the client side. 5.7.1 Resolution Limitation One big issue that came up while testing our implementation is that we were only able to perform tests with a resolution of 3840 ×1920 with usable results. On the one hand because CPU reached its limits when trying to increase the resolution to 7680 ×3840, as shown in the Table 5.3, even with the lowest upload rate possible. On the other hand because the QooCam 8K, which we used as the camera, was only able to provide those two resolutions, no tests with smaller resolutions were possible to showcase the difference in resource usage. This is also the reason why we were unable to test other resolutions on the client device, and all tests were performed there with 3840 ×1920. 5.7.2 Tiling Limitation We were able to find that the change in the tiling layouts to a multiple of cores available on the CPU does not have a big influence on the usage and the higher the amount of tiles, the lower the download rate on the client, as shown in the Table 5.4. However, unfortunately, we were unable to perform tests with four qualities of tiles, which the client application is theoretically capable of, as we reached the CPU limits even with three quality levels at 30 FPS. Still tests with three quality levels were possible with 24 FPS, to show the benefit of multiple quality levels. 5.7.3 FPS Limitation Another case that we were unable to test was the effect of different FPS settings, as our camera was only able to provide a stream with 24, 25 and 30 FPS, where our client devices are capable of displaying up to 120 FPS. It would have been interesting to see the utilization on the client devices with higher settings, but we were limited by the camera. 5.8. Use Cases 55 Another problem would also have been the CPU resources on the server, as it already reached its limits when running at 30 FPS and three tile qualities. 5.8 Use Cases Using 360°cameras to deliver live content combined with the tiled DASH approach presented in this thesis can be used in a wide range of applications. By making use of the transmission of individual tiles, only the region of interest is streamed in high resolution, while areas in the periphery are delivered at lower quality. This approach enables multiple users to have the same high quality live experience while having a significantly reduced bandwidth requirement. Following, some use cases will be presented, which illustrate that our approach can provide a flexible and efficient way to deliver 360°live videos across different fields of application. 5.8.1 Sports and Music Events Large scale live events such as football matches or music concerts, which take place in a fixed area observable from a single point of view, can be a good use case for live 360° video streaming. This would allow the end user to feel like being live in a stadium while the event takes place, and still have the possibility to decide where to concentrate at and explore the whole environment independently. Such flexibility is not possible in traditional broadcasts, which usually limits the viewers’ perspective to the one chosen by the camera operator. With the use of our tiled DASH streaming approach, it adapts the video quality to each individual viewer, allowing the best resolution in the current view while still having a low bandwidth requirement, as the surroundings are downloaded in lower qualities. This reduces the requirements for the client device, but especially for the streaming infrastructure. In large-scale events, where many end devices request data from the server, the server is relieved because only a fraction of the whole stream needs to be transmitted at the highest quality. 5.8.2 Education and Training Another big topic can be the creation of an immersive learning environment for student or apprentice training with the use of 360°cameras to deliver realistic scenarios. Live streaming surgeries would, for example, allow the medical student to observe complex procedures in high quality when focusing on the surgeon’s hands or the technical equipment necessary to monitor the patient in an operating room. By using our approach, the important parts the student is focusing on are always transferred in highest resolution, while relieving the network as the surroundings are transferred in lower quality. A similar use case is, for example, the apprentice training in industrial fields when showing them the operation of heavy machinery or the work in unsafe environments. 62 Chapter A. List of Acronyms HLS HTTP Live Streaming HMD Head-Mounted Display ISDN Integrated Services Digital Network ISO/IEC International Organization for Standardization ITU-R International Telecommunication Union((Radiocommunication Sector)) ITU-T International Telecommunication Union(Telecommunication Standardization Sector) KNN K-Nearest-Neighbors LL-DASH Low Latency DASH LL-HLS Low Latency HLS LR Linear Regression MPD Media Presentation Description MV-HEVC multi-view High Efficiency Video Coding NALU Network Abstraction Layer Unit NDK Native Development Kit POI Point of Interest PPS Picture Parameter Set PTZ Pan-Tilt-Zoom QP Quantization Parameter ROI Region Of Interest RTMP Real Time Messaging Protocol RTSP Real Time Streaming Protocol SDK Software Development Kit SHVC scalable High Efficiency Video Coding SIMD Single Instruction, Multiple Data SNR Signal-to-Noise Ratio SotA State-of-the-Art SPS Sequence Parameter Set SRD Spatial Relation Description TCP Transmission Control Protocol tmpfs Temporary File System UDP User Datagram Protocol VOD Video On Demand VPS Video Parameter Set VR Virtual-Reality WPP Wavefront Parallel Processing BIBLIOGRAPHY 63 Bibliography [1] Yixuan Ban, Lan Xie, Zhimin Xu, Xinggong Zhang, Zongming Guo, and Yue Wang. Cub360: Exploiting cross-users behaviors for viewport prediction in 360 video adaptive streaming. In 2018 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6, 2018. (page xv, 4) [2] Susmija Jabbireddy, Xuetong Sun, Xiaoxu Meng, and Amitabh Varshney. Foveated rendering: Motivation, taxonomy, and research directions, 2022. (page xv, 4, 5) [3] Cyril Concolato, Jean Le Feuvre, Franck Denoual, Fr´ed´eric Maz´e, Eric Nassor, Na¨el Ouedraogo, and Jonathan Taquet. Adaptive streaming of hevc tiled videos using mpeg-dash. IEEE Transactions on Circuits and Systems for Video Technology, 28(8):1981–1992, 2018. (page xv, 6) [4] Huixiang Yang, Hanlin Pan, and Lin Ma. A review on software defined content delivery network: A novel combination of cdn and sdn. IEEE Access, 11:43822–43843, 2023. (page xv, 7) [5] Eetu Latja. Parallel acceleration of h.265 video processing, 2017. (page xv, 8, 10) [6] Michal Podpora, Grzegorz Korba´s, and Aleksandra Kawala-Sterniuk. Yuv vs rgb – choosing a color space for human-machine interaction. Annals of Computer Science and Information Systems, Vol. 3, 09 2014. (page xv, 11, 12) [7] Abid Yaqoob, Mohammed Amine Togou, and Gabriel-Miro Muntean. Dynamic viewport selection-based prioritized bitrate adaptation for tile-based 360°video streaming. IEEE Access, 10:29377–29392, 2022. (page 3) [8] Zhi Pang. Vatp360: Viewport adaptive 360-degree video streaming based on tile priority. ArXiv, abs/2307.15984, 2023. (page 3) [9] Lan Xie, Zhimin Xu, Yixuan Ban, Xinggong Zhang, and Zongming Guo. 360probdash: Improving qoe of 360 video streaming using tile-based http adaptive streaming. In Proceedings of the 25th ACM International Conference on Multimedia, MM ’17, page 315–323, New York, NY, USA, 2017. Association for Computing Machinery. (page 3) [10] Ching-Ling Fan, Jean Lee, Wen-Chih Lo, Chun-Ying Huang, Kuan-Ta Chen, and Cheng-Hsin Hsu. Fixation prediction for 360°video streaming in head-mounted virtual reality. In Proceedings of the 27th Workshop on Network and Operating Systems Support for Digital Audio and Video, NOSSDAV’17, page 67–72, New York, NY, USA, 2017. Association for Computing Machinery. (page 3) 64 [11] Feng Qian, Lusheng Ji, Bo Han, and Vijay Gopalakrishnan. Optimizing 360 video delivery over cellular networks. In Proceedings of the 5th Workshop on All Things Cellular: Operations, Applications and Challenges, ATC ’16, page 1–6, New York, NY, USA, 2016. Association for Computing Machinery. (page 3) [12] Lili Wang, Xuehuai Shi, and Yi Liu. Foveated rendering: A state-of-the-art survey. Computational Visual Media, 9(2):195–228, 2023. (page 4) [13] Gary Sullivan, Jens-Rainer Ohm, and Thomas Wiegand. Overview of the high efficiency video coding (hevc) standard. Circuits and Systems for Video Technology, IEEE Transactions on, 22, 12 2012. (page 5, 8) [14] Iraj Sodagar. The mpeg-dash standard for multimedia streaming over the internet. IEEE MultiMedia, 18(4):62–67, 2011. (page 5) [15] Omar A. Niamut, Emmanuel Thomas, Lucia D’Acunto, Cyril Concolato, Franck Denoual, and Seong Yong Lim. Mpeg dash srd: spatial relationship description. In Proceedings of the 7th International Conference on Multimedia Systems, MMSys ’16, New York, NY, USA, 2016. Association for Computing Machinery. (page 5) [16] Jean Le Feuvre. Gpac filters. In Proceedings of the 11th ACM Multimedia Systems Conference, MMSys ’20, page 249–254, New York, NY, USA, 2020. Association for Computing Machinery. (page 6) [17] Waris Ali, Chao Fang, and Akmal Khan. A survey on the state-of-the-art cdn architectures and future directions. Journal of Network and Computer Applications, 236:104106, 2025. (page 7) [18] Akamai Technologies. Akamai Content Delivery Network, 2024. Accessed: 2025-0731. (page 7) [19] Cloudflare, Inc. Cloudflare CDN Documentation, 2025. Accessed: 2025-07-31. (page 7) [20] Amazon Web Services. Amazon CloudFront Documentation, 2025. Accessed: 202507-31. (page 7) [21] Amazon Web Services. AWS Documentation, 2025. Accessed: 2025-07-31. (page 7) [22] Microsoft Corporation. Azure Content Delivery Network (CDN), 2023. Accessed: 2025-07-31. (page 7) [23] Igor Sysoev. Nginx: a high-performance http server and reverse proxy. https: //nginx.org/, 2015. Accessed: 2025-03-25. (page 7, 23) [24] Apache Software Foundation. Apache Traffic Server Project, 2025. Accessed: 202507-31. (page 7) BIBLIOGRAPHY 65 [25] Al-Mukaddim Khan Pathan, Rajkumar Buyya, et al. A taxonomy and survey of content delivery networks. Grid computing and distributed systems laboratory, University of Melbourne, Technical Report, 4(2007):70, 2007. (page 7) [26] Iain E Richardson. The H. 264 advanced video compression standard. John Wiley & Sons, 2011. (page 7) [27] Michael Marcellin, Michael Gormish, Ali Bilgin, and Martin Boliek. An overview of jpeg-2000. pages 523 – 541, 02 2000. (page 8) [28] K Ramamohan Rao and Ping Yip. Discrete cosine transform: algorithms, advantages, applications. Academic press, 2014. (page 8) [29] A. Gersho. Quantization. IEEE Communications Society Magazine, 15(5):16–16, 1977. (page 8) [30] A Murat Tekalp. Digital video processing. Prentice Hall Press, 2015. (page 8) [31] ITU-T. RecommenAdation H.120: Codecs for Videoconferencing Using Primary Digital Group Transmission. Technical report, International Telecommunication Union, 1984. Standard for early digital video conferencing. (page 9) [32] ITU-T. Recommendation H.261: Video Codec for Audiovisual Services at p ×64 kbit/s. Technical report, International Telecommunication Union, 1990. First widely adopted video coding standard. (page 9) [33] ISO/IEC. ISO/IEC 11172: Coding of Moving Pictures and Associated Audio for Digital Storage Media at up to about 1.5 Mbit/s. Technical report, International Organization for Standardization, 1993. First standard for digital video storage (Video CD). (page 9) [34] ISO/IEC. ISO/IEC 13818: Generic Coding of Moving Pictures and Associated Audio Information. Technical report, International Organization for Standardization, 1995. Standard for digital television and DVD encoding. (page 9) [35] ITU-T. Recommendation H.263: Video Coding for Low Bitrate Communication. Technical report, International Telecommunication Union, 1995. Designed for video conferencing and early mobile video applications. (page 9) [36] ISO/IEC. ISO/IEC 14496-2: Coding of Audio-Visual Objects, Part 2: Visual. Technical report, International Organization for Standardization, 1998. First object-based video compression standard. (page 9) [37] ITU-T. Recommendation H.264: Advanced Video Coding for Generic Audiovisual Services. Technical report, International Telecommunication Union, 2003. Highly efficient video compression standard used in Blu-ray and streaming services. (page 9) 66 [38] ISO/IEC. ISO/IEC 23008-2: High Efficiency Video Coding (HEVC). Technical report, International Organization for Standardization, 2015. Successor to H.264, enabling higher compression efficiency and support for 4K/8K video. (page 9) [39] Benjamin Bross, Jianle Chen, Jens-Rainer Ohm, Gary J. Sullivan, and Ye-Kui Wang. Developments in international video coding standardization after avc, with an overview of versatile video coding (vvc). Proceedings of the IEEE, 109(9):1463–1493, 2021. (page 9) [40] Y. Chen, A. Norkin, C. F. Lam, and J. R. Ohm. An overview of core coding tools in the av1 video codec. IEEE Transactions on Circuits and Systems for Video Technology, 30(1):389–404, 2020. Royalty-free open-source codec developed by the Alliance for Open Media. (page 10) [41] J. Bankoski, P. Wilkins, and Y. Xu. Technical overview of vp8, an open-source video codec. In ACM SIGMM Conference on Multimedia Systems, 2011. Google’s opensource alternative to H.264. (page 10) [42] D. Mukherjee, J. Bankoski, A. Grange, R. Bultje, J. Han, and Y. Xu. A technical overview of vp9—the latest open-source video codec. SMPTE Motion Imaging Journal, 124(1):44–54, 2015. Google’s open-source codec competing with HEVC. (page 10) [43] Kiran Misra, Andrew Segall, Michael Horowitz, Shilin Xu, Arild Fuldseth, and Minhua Zhou. An overview of tiles in hevc. IEEE Journal of Selected Topics in Signal Processing, 7(6):969–977, 2013. (page 10) [44] Stefan Radicke, Jens-Uwe Hahn, Christos Grecos, and Qi Wang. A multi-threaded full-feature hevc encoder based on wavefront parallel processing. In 2014 International Conference on Signal Processing and Multimedia Applications (SIGMAP), pages 90– 98. IEEE, 2014. (page 10) [45] Jill M Boyce, Yan Ye, Jianle Chen, and Adarsh K Ramasubramonian. Overview of shvc: Scalable extensions of the high efficiency video coding standard. IEEE Transactions on Circuits and Systems for Video Technology, 26(1):20–34, 2015. (page 11) [46] Gerhard Tech, Ying Chen, Karsten M¨uller, Jens-Rainer Ohm, Anthony Vetro, and Ye-Kui Wang. Overview of the multiview and 3d extensions of high efficiency video coding. IEEE Transactions on Circuits and Systems for Video Technology, 26(1):35– 49, 2016. (page 11) [47] Adrian Ford and Alan Roberts. Colour space conversions. Westminster University, London, 1998(1-31):657, 1998. (page 11) BIBLIOGRAPHY 67 [48] Thomas Porter and Tom Duff. Compositing digital images. In Proceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’84, page 253–259, New York, NY, USA, 1984. Association for Computing Machinery. (page 11) [49] Etienne Dupuis. Optimizing yuv-rgb color space conversion using intel’s simd technology, 2003. (page 12) [50] B. Gordon, N. Chaddha, and T.H.-Y. Meng. A low-power multiplierless yuv to rgb converter based on human vision perception. In Proceedings of 1994 IEEE Workshop on VLSI Signal Processing, pages 408–417, 1994. (page 12) [51] Roshan Koju and S.R. Joshi. Comparative analysis of color image watermarking technique in rgb, yuv, and ycbcr color channels. Nepal Journal of Science and Technology, 15, 02 2015. (page 12) [52] H. Schulzrinne, A. Rao, and R. Lanphier. Rfc2326: Real time streaming protocol (rtsp), 1998. (page 13) [53] H. Schulzrinne, A. Rao, R. Lanphier, M. Westerlund, and M. Stiemerling. Rfc 7826: Real-time streaming protocol version 2.0, 2016. (page 13) [54] Iv´an Santos-Gonz´alez, Alexandra Rivero-Garc´ıa, Tom´as Gonz´alez-Barroso, Jezabel Molina-Gil, and Pino Caballero-Gil. Real-time streaming: A comparative study between rtsp and webrtc. In Carmelo R. Garc´ıa, Pino Caballero-Gil, Mike Burmester, and Alexis Quesada-Arencibia, editors, Ubiquitous Computing and Ambient Intelligence, pages 313–325, Cham, 2016. Springer International Publishing. (page 14) [55] Himanshu Parmar, Ed M Thornburgh, and Ed Adobe. Adobe’s real time messaging protocol, 2012. (page 14) [56] YouTube, LLC. YouTube Live, 2025. Accessed: 2025-09-12. (page 14) [57] Twitch Interactive, Inc. Twitch, 2025. Accessed: 2025-09-12. (page 14) [58] Meta Platforms, Inc. Facebook Live, 2025. Accessed: 2025-09-12. (page 14) [59] R. Pantos and W. May. Rfc 8216: Http live streaming, 2017. (page 14) [60] International Organization for Standardization. Information technology – Dynamic adaptive streaming over HTTP (DASH) – Part 1: Media presentation description and segment formats, 2014. (page 14) [61] Netflix, Inc. Netflix, 2025. Accessed: 2025-09-12. (page 15) [62] Amazon.com, Inc. Amazon Prime Video, 2025. Accessed: 2025-09-12. (page 15) 68 [63] International Organization for Standardization. Information technology – Dynamic adaptive streaming over HTTP (DASH) – Part 1: Media presentation description and segment formats, 2019. (page 15) [64] Apple Inc. HTTP Live Streaming (HLS) Specification, 2nd Edition, 2019. Accessed: 2025-03-25. (page 15) [65] Abdelhak Bentaleb, Zhengdao Zhan, Farzad Tashtarian, May Lim, Saad Harous, Christian Timmerer, Hermann Hellwagner, and Roger Zimmermann. Low latency live streaming implementation in dash and hls. In Proceedings of the 30th ACM International Conference on Multimedia, MM ’22, page 7343–7346, New York, NY, USA, 2022. Association for Computing Machinery. (page 15) [66] Mattis Jeppsson, H˚avard Espeland, Tomas Kupka, Ragnar Langseth, Andreas Petlund, Peng Qiaoqiao, Chuansong Xue, Konstantin Pogorelov, Micheal Riegler, Dag Johansen, Carsten Griwodz, and P˚al Halvorsen. Efficient live and on-demand tiled hevc 360 vr video streaming. In 2018 IEEE International Symposium on Multimedia (ISM), pages 81–88, 2018. (page 17) [67] Y. S´anchez De La Fuente, R. Skupin, and T. Schierl. Video processing for panoramic streaming using hevc and its scalable extensions. Multimedia Tools Appl., 76(4):5631–5659, February 2017. (page 17) [68] Sergo Martirosov, Marek Bureˇs, and Tom´aˇs Z´ıtka. Cyber sickness in low-immersive, semi-immersive, and fully immersive virtual reality. Virtual Reality, 26:15–32, 2022. (page 17) [69] Ubuntu. Ubuntu 22.04 lts (jammy jellyfish). https://ubuntu.com/download/ desktop, 2022. Accessed: 2025-03-25. (page 23) [70] Marko Viitanen, Ari Koivula, Ari Lemmetti, Arttu Yl¨a-Outinen, Jarno Vanne, and Timo D. H¨am¨al¨ainen. Kvazaar: Open-source hevc/h.265 encoder. In Proceedings of the 24th ACM International Conference on Multimedia, 2016. (page 23) [71] Fabrice Bellard and FFmpeg Developers. Ffmpeg: A complete, cross-platform solution to record, convert, and stream audio and video. https://ffmpeg.org/, 2000. Accessed: 2025-03-25. (page 23) [72] International Organization for Standardization. Information technology – Generic coding of moving pictures and associated audio information – Part 7: Advanced Audio Coding (AAC), 2006. (page 23) [73] Unity Technologies. TextureFormat.YUY2 (scripting api). https://docs.unity3d. com/ScriptReference/TextureFormat.YUY2.html, 2025. Accessed: 2025-09-03. (page 41) BIBLIOGRAPHY 69 [74] Kandao. Qoocam 8k enterprise, 2020. accessed 2025-08-13. (page 43) [75] Hisham Muhammad. htop, 2004. Accessed 2025-08-13. (page 44) [76] Paul Warren. iftop, 2002. Accessed 2025-08-13. (page 48)