Toward One-Second Latency:
Evolution of Live Media Streaming

Let us start with the general definition of streaming. Streaming is the continuous transmission of media such as video, audio and metadata from a server to a client and its simultaneous consumption by the client. In this definition, the critical term is “simultaneous” as we would not call it streaming if one downloaded a media file and played it after the download was completed (this would be instead called download-and-play). There are two implications here. First, the transmission rate must (loosely or tightly) match the consumption rate in order to provide uninterrupted playback. That is, the client must not run out of data (a situation referred to as buffer underrun), where the playback would rebuffer, or accept more data than it can hold in its buffer (a situation referred to as buffer overrun), where any excess data would be discarded. Second, the client’s consumption rate is limited by not only the bandwidth availability but also the real-time constraints. That is, the client cannot fetch or the server cannot push the media that is not available (i.e., captured, encoded and packaged) yet.

In this survey, live streaming refers to streaming not only live content but also linear and personal broadcasts, and surveillance videos. Live content includes any event or program happening in real time, such as live sports and live news programming. Linear broadcasts include programs on a TV channel that consist of live (e.g., live sports) as well as pre-recorded (e.g., movies and TV shows) content. In the case of linear broadcast, liveness in live streaming refers to the real-time streaming of the broadcast content — not necessarily to the nature of the content. On the other hand, personal broadcasts include live captured content from personal devices such as mobile phones and gaming consoles, and surveillance videos include live video captured by public safety, security and traffic cameras.

A viewer watching a football game, or breaking news, would be an example of live streaming consumption. One critical constraint in live streaming is the end-to-end (E2E) latency (also termed glass-to-glass latency), which is defined as the time gap between the capture of media (from a camera lens or microphone) and its playback (through a display or speaker).

How much latency is acceptable depends on the use case and application requirements. Fig. 1 depicts the continuum of different E2E latency values that are achievable or required by different applications, platforms and streaming configurations.

Based on today’s common understanding within the industry, we divide OTT live streaming into five broad categories based on the target E2E latency, noting that this categorization is subject to change based on future practices:

1. (a)

   High Latency: E2E latency is 45 or more seconds. The common application of this category is the one-way streaming of live events to large audiences where latency is relatively less important than the scalability and robustness of the service offering.

2. (b)

   Typical Latency: E2E latency ranges between 10 and 45 seconds. This latency range is commonly achieved by the majority of OTT services using HTTP adaptive streaming (HAS). It is also used as a fallback state when some components of the E2E media delivery pipeline do not have the necessary support for low-latency live streaming.

3. (c)

   Low Latency: E2E latency ranges between one and 10 seconds. This level of latency is essential to have a user experience similar to traditional broadcast TV. Example event types for this category include premium live sports, financial news and eSports. Another example application is second-screen experiences, where an event can simultaneously be consumed on different devices.

4. (d)

   Ultra-Low Latency: E2E latency is a second or less. Interactive experiences such as live commentary, betting, in-game wagering, online gambling and auctioning are the main drivers in this category. Ultra-low-latency live streaming introduces new challenges because this level of latency is on the same scale as commonly observed network latency variations, often due to bufferbloat [131] and other network anomalies such as packet reordering.

5. (e)

   Near-Real-Time Latency: E2E latency is in the order of tens of, up to 100, milliseconds. This level of latency is essential for networked applications that incorporate live streaming but also require specific interactivity or control-feedback features. Videoconferencing, cloud gaming, remote control of hardware platforms like drones, vehicles or surgical robots and metaverse are example applications.

We will survey the technologies that enable the latency ranges (a)–(c) above. However, our primary focus will be the low-latency live (LLL) streaming applications, where the key goal is to keep the latency below 10, preferably five, seconds. In this survey, however, we will not go into the details of the protocols used to achieve ultra-low (d) and near-real-time (e) latencies. Instead, Section V briefly touches on these topics and the preliminary discussions taking place in the new working group (called Media over QUIC or MOQ in short) in the Internet Engineering Task Force (IETF).

Today, an E2E typical latency of 10–45 seconds is practically achievable by the majority of live streaming services using HAS. However, such latency range is not low enough for certain content or use cases [42]. To support low latency, extensions for the popular HAS methods, namely Dynamic Adaptive Streaming over HTTP (DASH) and HTTP Live Streaming (HLS), have been developed and they are known as Low-latency DASH (LL-DASH) [51] and Low-latency HLS (LL-HLS) [14]. Details of these HAS methods and their low-latency extensions are presented in Sections II-D1 and IV.

Several recent technological advances have laid out the foundation for LLL streaming. Nevertheless, significant limitations and challenges still exist. Deploying an LLL streaming service requires a diligent examination and streamlining of every component in the media delivery pipeline. To this end, it is imperative to understand all the components and functionality that contribute to the E2E latency. Examining all the latency sources is one of the main goals of this survey.

Two important time-related metrics in live streaming are the startup delay and the latency, which are distinct from each other. The startup delay is the time lag from when the client joins a live stream (i.e., the viewer presses Play) until it begins to play out the media, while the latency is the time lag from when a media frame is captured until the same frame is played out. As the name implies, the startup delay only impacts the beginning of a stream. However, the latency persists throughout the whole playback and may sometimes increase or decrease in parts of the playback. These metrics are also independent of each other. That is, the startup delay might be quite small or large, while the latency could be low or high.

The startup delay and latency are both influential metrics. Together with rebuffering events and the playback speed, they can significantly impact the viewer’s quality of experience (QoE). The viewer QoE is a terminology that is introduced to describe a viewer’s perception and it reflects how satisfied or dissatisfied the viewer is while watching a stream based on various factors. There are two essential techniques to evaluate the viewer QoE: subjectively or objectively. A subjective QoE evaluation is performed through a user study, while an objective QoE evaluation is based on mathematical models such as ITU P.1203 (an open-source implementation is available in [147]) or the Yin model [186], which typically consider a weighted combination of the selected QoE factors. The eventual QoE value for a streaming session is generally a value in the same range as a mean opinion score (MOS, ITU P.800.1), which is between one (‘bad’) and five (‘excellent’).

Achieving a higher QoE often requires streaming higher-quality content with fewer rebuffering events. Higher-quality content means using a higher encoding bitrate due to higher resolution and/or frame rate, wider color gamut, etc. It may also mean a longer encoding time to allow the encoder to compress the media better using a longer look-ahead buffer or a multi-pass encoding approach. However, the time budget allocated for encoding is severely limited in the case of LLL streaming (as opposed to on-demand streaming). Thus, the encoders need to be well-designed for low-latency operation and the number of bitrate-resolution pairs (also known as the bitrate ladder) offered to the streaming clients should be carefully selected. On the other hand, eliminating or at least reducing the occurrences of rebuffering events, which are common due to transient network conditions, is easier with more media being buffered in the playback buffer. However, there is a limit to this, too, as buffering more media increases the latency unless the streaming client employs an adaptive playback speed controller to keep the latency under control while reducing the risk of rebuffering. In Section VII, we demonstrate how all these different experience metrics can be combined to compute a single QoE value.

The primary focus of this survey is ($i$) to provide a comprehensive anatomy of the components in live and low-latency live streaming systems, and how these components affect E2E latency, and ($ii$) to describe the state-of-the-art techniques and enhancements that have been proposed to date to achieve robust playback at low latencies. The survey is structured as follows:

In Section II, we first introduce the generic architecture of today’s live streaming systems and describe the different stages within this architecture, namely media contribution, ingest and distribution. Then, we list the protocol options for each stage and discuss their pros and cons. Specifically, we cover the push-based protocols for the contribution and pull-based and peer-to-peer (P2P) protocols for the distribution stages.

In Section III, we dissect the E2E latency into three serial parts for a more organized examination: media content preparation, delivery and consumption latency. We provide an in-depth analysis of the components contributing to the latency for each part. Then, we highlight improvements for each component to meet the LLL streaming requirements.

In Section IV, we introduce two key technologies enabling LLL streaming: chunked encoding/packaging and chunked delivery. We use the Common Media Application Format (CMAF) standard and HTTP/1.1’s chunked transfer encoding mode as examples. Then, we present the low-latency extensions for the existing HAS protocols, namely LL-DASH and LL-HLS, and show how the same chunked-encoded media can be efficiently served using these two extensions. The section then continues with a discussion on the High Efficiency Streaming Protocol (HESP).

In Section V, we present a brief overview of the developments in using real-time communication protocols together with DASH to improve interactivity in LLL streaming. We also summarize the preliminary discussions that led to forming of the new Media over QUIC (MOQ) working group in the IETF.

In Section VI, we discuss the main components of a media player and discuss bandwidth measurement, rate adaptation (adaptive bitrate, ABR), playback speed control and (playback) buffer-management algorithms designed for low latency. The section also provides an evaluation of the existing open-source media player implementations.

In Section VII, we present the summary and findings of Twitch’s grand challenge on low-latency live streaming [168]. This summary illustrates the best practices to prototype an E2E LLL streaming solution and test the individual components in the delivery workflow.

Finally, Section VIII concludes the survey by summarizing the lessons learned and future directions for LLL streaming.

The list of the abbreviations used frequently in this survey is given in Table I.

While we cover many topics on low-latency live streaming in this survey, the following subject areas (for which we provide references for the interested readers) are out-of-scope:

• •

  Two-way, interactive videoconferencing applications: [39, 114, 90, 145].

• •

  Cloud gaming [57, 3, 70, 36].

• •

  QoE management, measurement and monitoring: [77, 6, 20, 171, 151].

• •

  Stream manipulation, content steering and ad insertion: [54, 45, 15, 16, 155].

A typical E2E live streaming architecture is shown in Fig. 2, consisting of three main stages: media contribution, media ingest and media distribution. The contribution stage involves content acquisition and production. As the content is acquired from a source (e.g., via cameras at a studio or stadium), it is first encoded by a contribution encoder to produce very high-quality output at a high bitrate. The output of the contribution encoder is then input to a multi-bitrate distribution encoder (also called an OTT encoder or transcoder). For the transmission from the contribution encoder to the distribution encoder, one can use a specialized media transport protocol (see Section II-B for some examples). One can also use plain multicast if there are multiple distribution encoders running in parallel for load balancing or redundancy purposes.

The second (ingest) stage involves content packaging, encryption and transmission to an origin server. Out of the incoming high-quality contribution feed, the distribution encoder generates multiple representations (at different bitrates and resolutions using one or more codecs) according to a preset bitrate ladder. It then packages them into one or more delivery formats, such as DASH and HLS, and optionally applies encryption. The packaged content that includes all the DASH/HLS media segments and the associated manifest files is then ingested to an origin server. There are many options for the ingest protocol, some of which are described in Section II-C.

Finally, the distribution stage involves content delivery and consumption. The packaged content is distributed from the origin server to media players connected through different last-mile access networks. When using HTTP for distribution, one or more content delivery networks (CDN) can be used to shield the origin server and scale the distribution to millions of media players around the globe. As is the case with the other stages, there are many options one can use for distributing the content. Some examples are provided in Section II-D.

The choice and configuration of the various components, such as the contribution and distribution encoders, packagers, contribution, ingest and distribution protocols, as well as rate-adaptation, buffer-management and other playback-related algorithms running on the media player, all affect the E2E latency and robustness of the playback. In the remainder of this section, we summarize various protocol options for media contribution, ingest and distribution, along with their advantages and disadvantages.

Protocols used for media contribution are generally push-based protocols where the server/encoder pushes the media. A few traditional protocols were designed in the early days of streaming over the Internet and have been in use for many years. Nonetheless, the last decade has seen an emergence of new standards-based and proprietary protocols. These were designed for reliable, low-latency live media transmission over best-effort (i.e., unmanaged) networks. The end goal for these protocols has been to reduce operational costs and increase deployability. We summarize these protocols below.

• •

  Real-Time Messaging Protocol (RTMP) (RFC 7425) was created by Adobe as a delivery protocol for streaming video and audio over the Internet. It uses a persistent TCP connection to transmit media data from a source to a single receiver with Adobe Flash Player support. It was initially designed for both video contribution and distribution. However, because Flash was ($i$) a proprietary technology, ($ii$) increasingly vulnerable to cyber attacks, and ($iii$) deprecated and not allowed to work in modern browsers, playing media over RTMP is no longer supported on many end-points. While Adobe announced the end-of-life for its Flash Player back in 2017 [4], RTMP remains popular and many content platforms continue to use it for media contribution.

• •

  Real-Time Transport Protocol (RTP) (RFC 3550) is a protocol developed for the real-time transmission of multimedia data in unicast or multicast mode. Although it is not required, RTP typically runs over UDP. Through its header, RTP supports payload type identification, sequence numbering and timestamping. RFC 3550 also defines the RTP Control Protocol (RTCP), whose primary purpose is to provide applications with minimal control and identification functionality. Additionally, RTCP provides a scalable monitoring service for RTP transport by using sender, receiver and extended reports (RFC 3611). While RTP is widely used as a standalone protocol in many multimedia applications, for media contribution, it is used as part of another protocol or standard, such as described below.

• •

  SMPTE ST 2022 [157] is a suite of standards developed by the Society of Motion Picture and Television Engineers (SMPTE) that focuses on professional quality video delivered over IP. The suite currently consists of eight standards. The ST 2022-2, 2022-3 and 2022-4 standards describe the carriage of MPEG-2 Transport Streams over IP. The ST 2022-1 and 2022-5 standards define a forward error correction technique for the carriage of real-time compressed and uncompressed (termed as High Bit Rate (HBR), 270 Mbps or higher) video/audio over IP, respectively. The ST 2022-6 standard defines a unidirectional IP-based protocol for the transport of real-time uncompressed video, audio and ancillary data typically found on serial digital interfaces (SDI). The ST 2022-7 standard is for protecting a video stream against data loss by seamlessly switching between multiple redundant streams of RTP packets. Finally, the ST 2022-8 standard specifies the use of and constraints for ST 2022-6 streams in conjunction with the timing model of ST 2110-10.

• •

  Secure Reliable Transport (SRT) [75] is an open-source protocol first released in 2013 to provide end-to-end security, resiliency and dynamic end-point adjustment based on real-time network conditions while delivering high-quality video. SRT has been shown to have high robustness against packet loss, jitter, latency and variable bandwidth. The development of SRT is managed by the SRT Alliance, which currently has 400+ members [74]. SRT runs over UDP and is primarily used for contribution, although some projects are underway to use it for distribution. In that case, due to the lack of support and integration in browsers, an SRT gateway will be needed to convert the incoming SRT stream into a format the browsers can use.

• •

  Reliable Internet Stream Transport (RIST) [172] is a standard developed by the Video Services Forum (VSF). It is a reliable, low-latency video transmission solution intended to be used over unmanaged networks. RIST uses RTP transport for the media packets where any missing packet is reported using RTCP feedback (RFC 4585) and then retransmitted using RTP retransmissions. RIST has multiple profiles defined so far (Simple, Main and Advanced) with different feature sets. RIST is also available in open source and has been implemented in many popular media software packages.

• •

  Zixi [189] is a proprietary congestion and network-aware protocol designed for media transmission over best-effort networks. It dynamically adjusts to varying bandwidth and includes a set of forward error correction (FEC) techniques to ensure reliable and low-latency delivery.

With some exceptions, requirements for the media ingest protocols are similar to those of the media contribution protocols. For that reason, most of the contribution protocols (RTMP, SRT, RIST, Zixi, etc.) can be used for media ingest with no or little changes. On the other hand, there are ingest protocols developed for HTTP-based workflows. There are also those proposed for personal live broadcasts (i.e., there is not necessarily a media contribution stage). We summarize these protocols below.

• •

  Microsoft Smooth Streaming Live Ingest is a legacy media ingest protocol developed by Microsoft as part of the Smooth Streaming product suite [121]. This protocol was widely used for both on-demand and live content for several years. With the advances in media codecs, distribution protocols and signaling of metadata and timed text, this protocol was later replaced by Microsoft’s Azure Media Services fragmented MP4 Live Ingest protocol [120]. This new protocol supports service/encoder failover and redundancy to improve the robustness of the ingested media.

• •

  DASH-IF Live Media Ingest Protocol [52] is a specification developed through the collaboration of many vendors and service providers. The specification defines two interfaces. Both interfaces use the HTTP POST (or PUT) method to transmit media objects from an ingest source to a receiving entity. The first interface describes the direct ingest of media formatted as a CMAF presentation to a receiving entity. The receiving entity, a packager or a cloud media service, can produce DASH and HLS segments along with the manifests with optional transcoding, stitching, (re)encrypting and watermarking. In the second interface, the media is already packaged in the DASH or HLS format. In this case, the media objects are transmitted to a receiving entity (such as an origin server or a cloud media service) along with the manifests. If desired, the receiving entity can do further manifest manipulation. Otherwise, it stores and serves the segments. Both interfaces support the carriage of video and audio data, timed metadata and timed text. The specification also presents the guidelines for synchronizing multiple ingest sources and discusses redundancy and failover behaviors. Some open-source tools are provided in [52].

• •

  Warp [113] is a segmented live media transport protocol proposed by Twitch and is currently under discussion in the IETF. Warp takes any live input and maps it to individual QUIC streams based on the specific encoding. By adjusting the transmission priorities, Warp tries to deliver as much media as possible, given the latency requirements. Warp can be used for both media ingest and distribution. An open-source implementation and demo are available at [112].

• •

  RUSH [143] is a bidirectional application-level protocol designed for live media ingest that runs on top of QUIC. RUSH’s primary goal is to replace RTMP with support for new codecs, message types and multi-track support. Similar to Warp, RUSH also provides provisions for timely (but not necessarily fully reliable) media delivery. RUSH can also be used for both media ingest and distribution.

• •

  WebRTC-HTTP Ingestion Protocol (WHIP) [128] is an HTTP-based protocol that allows Web Real-Time Communication (WebRTC)-based ingestion of media. A WHIP client producing and encoding live media sends an HTTP POST request to establish a session with the ingest end-point. Once the session is set up, media flows unidirectionally from the WHIP client to this ingest end-point, with no possibility of adding or removing media tracks. WHIP is currently an active working group document in the IETF and is close to publication. Several open-source implementations are also available (e.g.,  [62, 117, 116]).

Media contribution and ingest protocols often run in point-to-point mode, although they can also run in point-to-multipoint mode for load balancing and redundancy purposes. The top priority for these protocols is reliability, i.e., no-loss media transmission, since any data loss during media contribution or ingest introduces an artifact that will impact any viewer consuming this media content. In the case of live media, low-latency contribution and ingest also become important. On the other hand, media distribution protocols also run in mostly point-to-point mode. However, these protocols need to be able to scale from a few to millions of media players. While media distribution originates from an origin server, it is always shielded by other entities (caches, replication points, etc.) that distribute the content on behalf of the origin server. Next, we first present the details for using HTTP for media distribution (i.e., HTTP adaptive streaming, HAS), which is by far the most popular approach today. Then, we briefly discuss peer-to-peer (P2P) delivery and give examples.

The live content (or content from a linear source) must first be acquired and then prepared for OTT distribution. Beyond the acquisition latency, which is usually negligible, the MCPL comprises latency introduced in encoding, packaging and ingest, which are described below.

1. 1.

   The (contribution and distribution) encoding latency is often the largest contributor to the MCPL¹¹1Since video encoding is significantly more complex and takes a much longer time than audio encoding, we mainly focus on the latency due to video encoding in this section.. It is impacted by many factors, including the choice of codec, content complexity, target quality and encoding parameters (e.g., bitrate ladder, and group-of-pictures (GoP) and segment duration).

2. 2.

   The packaging latency is introduced because of the encapsulation of the encoded stream into one or more delivery formats. An optional encryption step may further increase this latency.

3. 3.

   The ingest latency is introduced when uploading the packaged media segments and the associated manifests to an origin server.

Next, we discuss the details of each latency component.

The MCDL is primarily due to two main tasks: CDN distribution and last-mile delivery.

1. 1.

   The CDN distribution latency is introduced when propagating the media segments through different caches of the CDN cloud.

2. 2.

   The last-mile delivery latency differs depending on which access network the viewer is connected to during streaming. Also, additional latency could be introduced depending on the geographical location and position of the closest CDN end-point.

Below, we discuss the details of each latency component.

MCCL is mainly introduced because of the many media player tasks such as rate adaptation, buffering, playhead positioning, license/key fetching, decryption, decoder priming, decoding and rendering. Media players are typically designed to ensure smooth playback, which is achieved by buffering several seconds of media. In LLL streaming, buffering is severely limited due to low-latency constraints. The media player is one of the more influential components contributing to latency, which may add up to 50% of the E2E latency if it is not tailored for LLL streaming. Hence, adapting the player for LLL streaming can go a long way [99].

Consider Fig. 7, which demonstrates a live streaming scenario where the content is packaged into four-second segments, and the numbers inside the boxes indicate the segment numbers. After downloading the manifest, a media player joins the live session 18 seconds into the content. There are three common strategies the media player can adopt (each of which contributes differently to the latency):

• •

  The media player requires at least $m$ segments to be downloaded fully before the playback can start. For the native (HLS) media player on iOS/iPadOS/tvOS platforms, $m$ equals three. The media player checks the playlist and determines that segments #1 through #4 are available so far. Consequently, it requests segments #2, #3 and #4. Assuming the segment fetching time is negligible (meaning that the RTT between the media player and the server is small and the available bandwidth is plenty), the media player can start the playback quickly but with a latency of 14 seconds.

• •

  The media player fetches the last fully available segment. In this case, the latency drops to six seconds.

• •

  The media player prudently defers sending the request for segment #4 at the time of joining, waits for two seconds and then sends the request for segment #5. This way, the media player achieves the lowest latency possible, at four seconds, although the startup delay increases by two seconds.

Downloading the media segments is not sufficient to start the decoding and rendering. The media player must also communicate with the licensing server, fetch the keys to perform decryption, and then prime the video and audio decoders. Any extra delay in this process may negatively affect the latency as well as the startup delay. Beyond this, there are other new challenges (in addition to those discussed above) for the media player where modules like bandwidth measurement, playback speed control, (playback) buffer management and bitrate selection scheme become non-trivial. These modules should be re-visited and re-designed to support streaming at low latency. We continue this discussion further in Section VI.

WebRTC [176, 109] is a set of W3C and IETF standards that delivers real-time content to viewers, typically with an end-to-end latency of under half a second. It builds upon the prior RTP protocol suite (see Section II-B). Support for WebRTC is built into all modern Web browsers across desktop and mobile devices, and the library functions allow for video, audio and data streaming. While the original focus of WebRTC was videoconferencing, it is increasingly being used today for real-time streaming of premium content because of its ultra-low latency features. These features may enable several new user experiences, especially those involving user interactivity that is not easy to deliver or even possible with traditional broadcast or HTTP-based streaming delivery protocols alone.

As interactive premium content services scale up, the integration of WebRTC with DASH is deemed essential. However, initial attempts at this highlighted gaps in WebRTC and DASH. In May 2022, the DASH-IF invited experts to study the details of a WebRTC-based premium streaming ecosystem. The goals were to recognize synergies between WebRTC and DASH, explore opportunities and identify open issues. This study resulted in an initial report [53] that outlined example use cases, developed key performance indicators and provided a baseline architecture. The report also outlined the technical work needed to integrate WebRTC with DASH. An initial proof-of-concept implementation based on the report’s findings is available in [109].

The IETF formed a new working group called Media over QUIC (MOQ) in 2022 to explore new opportunities for low-latency streaming using QUIC [83]. Its charter is to develop a scalable and efficient low-latency media delivery solution for media ingest and distribution in browser and non-browser environments. Targeted applications include live streaming, cloud gaming, remote desktop, videoconferencing and eSports. The work is still in its infancy, but media over QUIC running in an H3 or WebTransport [175] environment could be a game-changer [89].

MOQ defines a latency-configurable delivery protocol for sending media from one or more producer(s) to consumer(s) through relays using either WebTransport (in browsers) or raw QUIC (elsewhere). Relays take incoming media and forward it to one or more relays or consumers to scale media distribution without needing a separate encoding for each consumer. They decide what to send in what order or discard based on specific metadata exposed in the envelope of the incoming packets to respond to congestion and satisfy the application’s latency requirements. Consumers can also trade off quality with latency by choosing how long to wait for media based on their network conditions and desired user experience. Fig. 16 shows that MOQ is designed to be a standard media protocol stack, which can facilitate two primary functions: ($i$) enabling live streaming of events, news and sports with enhanced interactive capabilities, and ($ii$) scaling up real-time media applications to cater to larger audiences.

The MOQ proposals are still under discussion and the protocol details are still evolving.

In traditional non-LLL streaming, measuring bandwidth is straightforward. Consider Fig. 17 where the media player fetches the segments already packaged and available on the server. Therefore, the fundamental equation of bandwidth measurement ($Q_{i}/\tau_{i}$) provides a good approximation for the available bandwidth, where $Q_{i}$ denotes the size of segment $i$ and $\tau_{i}$ denotes the download time for segment $i$ (i.e., the time difference between when the request was sent and when the segment was fully received).

In contrast, when using chunked encoding/packaging and delivery for LLL streaming as depicted in Fig. 18, the media player fetches the live-edge segment where the chunks are transmitted in two different transmission phases [31, 32, 106]. First, in the network-limited phase, the available chunks of the requested segment are sent by the server (e.g., first the origin and then the CDN server) at the full network speed to the media player. Then, in the source-limited phase, the remaining chunks of the requested segment are sent to the media player as they become available. Assuming the network is fast enough, idle time is introduced between the chunks as they are sent from the server. This way, the chunks are sent with relatively more consistent timing ($\tau_{i}\sim D$). If the network is not fast enough, then the transmission is network-limited, and the ABR scheme naturally needs to downshift to a lower bitrate. In either case, the total segment download time, from the first chunk to the last chunk (including idle times in between), cannot be significantly shorter than its duration. Thus, a media player naively using the $Q_{i}/\tau_{i}$ formula will compute a value nearly equal to the segment encoding bitrate. This prevents the media player from switching to a higher bitrate level, even if the network has enough bandwidth to support a higher level.

To understand this better, refer to Fig. 19. After segment $i$ is fully downloaded, the media player uses the $Q_{i}/\tau_{i}$ formula to measure the bandwidth, where

Here, $q^{n}_{i}$ is the size of the $n^{th}$ chunk, $z$ is the number of chunks in segment $i$, and $b^{n}_{i}$ and $e^{n}_{i}$ are the download beginning and end times of the $n^{th}$ chunk, respectively. In (1), the segment download time ($\tau_{i}$) includes the inter-chunk idle periods, potentially resulting in incorrect bandwidth measurements.

A new chunk-aware bandwidth measurement technique taking the inter-chunk idle periods into account is clearly needed. The first solution in this space was [31], which proposed the ABR for Chunked Transfer Encoding (ACTE) method. ACTE used a per-chunk sliding window moving average (SWMA) bandwidth measurement algorithm and predicted the future bandwidth based on a recursive least squares (RLS) algorithm. For each segment download, measuring the bandwidth involved two steps: ($i$) calculating the download rate of each chunk, which equaled its size divided by this chunk’s end time minus the previous chunk’s end time, while ($ii$) filtering out the chunks with a non-negligible idle period (the chunks whose download rate was within $\pm$20% of the average segment download rate).

ACTE achieved good performance in case of short inter-chunk idle times. However, it suffered bandwidth underestimation problems when the idle times were longer. To rectify this issue, first, the Low-on-Latency (LoL) solution [106] and then its extension, LoL⁺ [25], were developed. The LoL and LoL⁺ solutions are learning-based and work efficiently for both short and long inter-chunk idle times, thanks to:

• •

  Chunk Boundary Identification: The module first successfully identifies each chunk’s beginning and end times by capturing the moof box in the downloaded data. It leverages the fact that the chunks are transmitted as separate HTTP chunks through CTE. Via the Fetch API, this module uses Streaming Response Body³³3Available [Online]: https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams, which allows tracking the progress of the chunk downloads and parsing the chunk payloads in real time. When a moof box is captured, the module stores the time as the beginning time of the chunk download. Then, it stores the end time of the chunk and its size (in bytes) when the chunk is fully downloaded. The inter-chunk idle times can be removed entirely from the segment download time with the accurate beginning and end times for each downloaded chunk.

• •

  Chunk Filtering: Upon receiving all the chunks of a segment, the filtering process is triggered to remove noisy measurements. In this case, the first and the last chunks are ignored in the bandwidth calculation to omit any transient outliers.

• •

  Segment Bandwidth Smoothing: After the filtering process is completed, this module first computes the segment download time ($\tau$) using only the chunks that passed the filtering process and then uses the SWMA-based smoothing algorithm to compute the measure of the bandwidth.

Later, [26] found that the LoL and LoL⁺ bandwidth measurement modules could sometimes perform poorly if the network conditions showed significant variations. To address such cases, the authors proposed the Automated Model for Prediction (AMP) that encompassed techniques for bandwidth prediction and model auto-selection designed explicitly for LLL streaming with CTE. The results from the AMP study showed that using several prediction models and automatically selecting a good one during the streaming session was beneficial. Different prediction models have their own characteristics and strengths, and hence, their effectiveness varies under different network conditions and user mobility types.

We note that since LL-HLS uses a separate request for each partial segment, the presented bandwidth measurement problem does not apply to LL-HLS. We also note that other bandwidth measurement methods exist, including the traditional active probing methods such as packet pair/train dispersion probing that is often used to estimate the end-to-end capacity of a network path [142, 18], and Will’s simple side-load (WSSL) method, which retrieves a fully available object (such as a previous segment) along with the live-edge segment for bandwidth measurement purposes [31]. However, active probing methods may be less desirable in media streaming applications as these methods introduce additional traffic to the network (which already is a scarce resource in streaming) [37]. They are often not well-adapted to handle streaming traffic’s periodic and bursty nature, either [173]. To mitigate these issues, [104] designed an active probing algorithm that probes using incremental data rate (instead of piggyback traffic) that backs off when network congestion is detected.

Traditional (non-LLL) ABR schemes [30] cannot sustain smooth playback with tiny playback buffers as they do not provide enough protection against sudden bandwidth drops. To be able to deploy LLL streaming services at scale, several considerations should be taken into account:

• •

  Small Buffer Levels: The duration of the media in the playback buffer should not be larger than the target latency. That means when the target latency is in the one-second range, media players have just a few hundred milliseconds to react to bandwidth variations. Since real-world network conditions can vary significantly, especially on mobile networks, media players are challenged with balancing the number of bitrate switches against preventing rebuffering.

• •

  Variable Playback Speeds: Changing playback speed during the live streaming session can be helpful to maintain the target latency. When the media player falls significantly behind the live edge (i.e., the instantaneous latency increases), it can accelerate the playback to catch up and prevent its buffer from growing arbitrarily as long as media arrives fast enough. However, in such cases, buffer-based ABR schemes like BOLA [159] may struggle since the buffer level may not grow large enough for upshifting.

• •

  Fairness or Controlled Unfairness: Media players watching the same stream (or a different stream from the same server) or the ones sharing network links will be competing for the available bandwidth unless the supply of the server and network capacity is guaranteed to be more than the demand. Previously, [9] showed that one or more problems, such as bitrate oscillations, network resource underutilization, sustained starvation and (bitrate) unfairness among the media players, were inevitable. If a media player is too greedy or behaves incorrectly, it risks degrading the experience of other players. On the other hand, not all media players run on devices with identical capabilities or display sizes, nor do they consume the same or the same type of content (e.g., sports, news or a movie). Thus, fair-share bandwidth allocation is largely a bad idea [27]. One rather needs controlled unfairness instead of absolute bitrate fairness not only to avoid bitrate oscillations but also to achieve quality fairness among the media players [22, 23].

• •

  Playback Synchronization: Many applications in live streaming rely on synchronized playback, such as synchronizing across multiple users co-watching the stream to maintain fairness and trust (and meet legal obligations in some cases) in betting applications and synchronizing across multiple camera angles of the same content that a user is consuming concurrently.

The recent ABR schemes designed explicitly for LLL streaming can be categorized as heuristic-based or learning-based. Heuristic-based schemes (e.g., LoL heuristic-based [106], Stallion [73] and Fleet [103]) use mathematical rules or approximations to determine the next best bitrate to request from the server. In an earlier study, Peng et al. [137] used the hybrid control theory to formulate the ABR decision problem for low latency. It implemented three essential rules: a heuristic-based playback rate control, latency-constrained bitrate control and QoE-oriented adaptive frame dropping. It also leveraged Kaufman’s adaptive moving average (KAMA) algorithm to predict the bitrate for the next segment for better rate adaptation.

In contrast, learning-based schemes (e.g., LoL learning-based [106], L2A [94], Fugu [184] and ALVS [134]) learn from changes in the system environment to make decisions based on past data. More specifically, leveraging advances in deep reinforcement learning (DRL), Jiang et al. [92], Hong et al. [78], Zhao et al. [188], and Wang et al. [178] proposed the following ABR schemes for LLL streaming: HD3, CBLC, L2AC and BitLat, respectively. HD3 [92] is a DRL algorithm that combines discrete actions (bitrate and target buffer level) with continuous actions (latency limit values) to optimize video quality while minimizing E2E latency. CBLC [78] uses a deterministic policy gradient for continuous ABR decisions and latency control tasks. L2AC [188] trains several separate DRL-based models on various network conditions with a reward function that maximizes QoE and penalizes latency. Then, it uses the model ensemble technique to combine these models to improve the overall performance. BitLat [178] develops a latency-aware RL-based solution to select a suitable bitrate while maintaining low latency. Further, it includes a dynamic reward method in the model to boost performance.

Each of these ABR schemes has its own merits in that it could achieve better performance in some scenarios or under certain network conditions but not in others. This suggests that having an ABR switching module that implements multiple ABR schemes and could select the best one based on the current requirements and conditions may help to leverage these schemes better [106]. In selecting a suitable ABR scheme for the job, one should consider the following aspects:

• •

  Predictability of the Media Characteristics and Network Conditions: Bitrate selection depends on how much certainty we have about the content being streamed and the conditions of the network to which the media player is connected. If chunk/segment sizes vary significantly over time and are not known a priori (within an error margin) or if the available bandwidth varies too much, and hence, becomes difficult to predict, then a learning-based scheme is a better choice as heuristic-based schemes may perform poorly with larger prediction errors. This arises from the inherent limitation of heuristic-based schemes, as they cannot consistently adapt to varying network conditions.

• •

  Significance of the QoE Model: In systems where the desired QoE model is not well-defined, a learning-based scheme is a better choice as it can individually learn from each QoE metric and make nuanced trade-offs. This contrasts with heuristic-based schemes, which rely heavily on the QoE model function.

• •

  Performance Comparison of the ABR Schemes: At times, the choice and performance of an ABR scheme may not be clear. Conducting extensive experiments to evaluate the performance of each ABR scheme under different conditions would be the ideal way of understanding their performances and determining which one to use. This approach is tedious but also needed as we explore the idea of ABR switching, as explained next.

• •

  Switching between the ABR Schemes: This is achieved through a data-driven AI-based switcher, which takes many inputs from the environment (e.g., network conditions, player status, stream complexity and LLL requirements) and auto-selects the best-performing ABR at runtime, and keeps switching between these ABR schemes during the streaming session for better performance.

Using the conventional playback speed control techniques without adapting them for LLL streaming or not using any such techniques may cause two major issues: more frequent rebuffering events (as the buffer sizes are smaller) and an uncontrolled increase in latency (as rebuffering durations keep adding up to the latency). The main goal of the playback speed control is to determine a rendering speed that ensures a latency close to the target while taking a calculated risk of rebuffering [50, 137, 11].

[25] developed a hybrid playback speed controller based on two variables: ($i$) the difference between the current and target latency, and ($ii$) the difference between the current playback buffer level and a pre-specified/configured safe threshold (all measured in units of time). Specifically, the media player considers the following cases when dynamically adjusting the playback speed:

1. Case 1:

   The current buffer level is below the safe threshold. In this case, the playback should be slowed down.

2. Case 2:

   The current buffer level is equal to or above the safe threshold and further:

   • 2a:

     The current latency is close enough to the target latency, then the media is played at the normal speed (1x).

   • 2b:

     The current latency is lower than the target latency, then the media is played at a slower speed.

   • 2c:

     The current latency is higher than the target latency, then the media is played at a faster speed.

Here, the playback speed is determined from a configurable speed range (e.g., $0.5-1.5\times$) based on the deviation between the current and target latencies and between the current buffer level and safe threshold. However, to reduce the impact of playback speed changes on the QoE, the controller should try to stay at the normal speed as long as possible. Table IV illustrates an example of the abovementioned cases.

Varying the playback speed must still consider the synchronization between the media streams sharing the same presentation timeline. For the video, the process is straightforward since frames can readily be displayed, either faster or slower than normal. For the audio, time-scale modification [133] can be applied to enable slower or faster playback while preserving the pitch of the signal. Note that all the media is still played out (no skips) despite the playback speed changes.

One problem with playback speed adaptation is that it may introduce discernible artifacts to the media, which could sometimes annoy the viewer (i.e., the viewer can tell that the video/audio has been sped up or slowed down). More sophisticated playback speed control solutions have been developed to mitigate this, such as the content-aware playback speed control (CAPSC) algorithms [10, 8] that intelligently vary the playback speed based on the content to be played at that instant such that the artifacts introduced are less noticeable (or almost unnoticeable) by the viewers.

LLL streaming requires robust buffer management that can tolerate sudden and significant changes in network conditions. Anytime, the playback buffer level should be above the given safe threshold. If one is provided, the buffer level should also be below the maximum latency acceptable for the service. This can be achieved by configuring the media player settings properly. The default settings for most media players are set to favor smoother playback with less rebuffering, meaning that they prioritize having more media data in the playback buffer over operating at a lower latency. In this section, we describe some guidelines for attuning the media player settings for LLL streaming:

• •

  Buffer Sizing and Initial Playhead Positioning: Media players, by default, are configured to download a specific number of segments (assuming a particular minimum segment duration of several seconds) and/or a specific number of seconds of media before the playback can commence. For example, DASH-based media players may be informed to download a specific duration of media through the minBufferTime attribute in the manifest [88].

  Recall that the startup delay is distinct from the latency (refer to Section I-C). The media player can start downloading from earlier segments to fill the necessary buffers and start the playback sooner. However, in this case, the latency will not be necessarily low. Alternatively, the media player can download a segment closer to the live edge to reduce the latency, but the startup delay may be larger this time. Therefore, one must coherently configure the media player’s settings regarding the startup delay, buffer sizing and latency and make sure that one setting does not override another.

• •

  Sticking to the Live Edge: When a media player stalls for a certain amount of time and then resumes the playback from where it left off, the instantaneous latency will at least increase by the rebuffering duration. By controlling the playback speed, the media player can attempt to close the gap to the live edge, provided the gap is not significant to start with. If it is, the media player can instead jump forward in the presentation timeline (i.e., by skipping media) to reduce the latency back to the target value. This behavior has been available in the dash.js reference player since v2.9.2.

• •

  Resilience to Late/Missing Segments: When a media player receives an HTTP 404 error for a requested segment, it can be due to three reasons. First, the media player might have sent the request earlier than it should have. Second, the CDN and the origin server may not have started receiving the requested segment because of an unexpected delay in the media contribution/ingest stages (refer to Fig. 2). Third, the segment might have been corrupted or lost in the upstream. Upon receiving the 404 error, the media player can wait for a bit and retry the request. Alternatively, it can request the corresponding segment from a different representation. If none of the temporally same segments is available, the media player must skip this segment and continue with the subsequent segment(s).

  If a media player occasionally or consistently keeps sending early requests, this causes a major problem in media distribution. Even after the particular segment becomes available on the origin server, the 404 error will be cached by downstream CDN servers, and all the media players requesting this segment will also receive the 404 error until the cached error message expires. In other words, a misbehaving or faulty media player could introduce a problem to all the other media players watching the same LLL stream. To avoid this avalanche effect, some CDNs turn off caching the 404 errors at the risk of increasing the request traffic back to the origin server. Another (but costly) solution is to implement the blocking request feature in the origin server, which is already required to support LL-HLS. On the LL-DASH side, the media players must be time-synchronized with the packagers that prepare the media segments and manifests. An NTP server (e.g., https://time.akamai.com/) can be signaled in the manifest (using the UTCTiming element), and this way, clock drifts can be avoided and the media players can be kept tightly synchronized.

  For both LL-DASH and LL-HLS, enabling truly fault-tolerant media contribution and media ingest stages is critical. This can be achieved by using multiple encoders, packagers and origin servers in the workflow so that multiple interchangeable and synchronized outputs can be produced. For details on the ongoing standardization, see [118, 127].

There are multiple open-source media players available. These players have support for DASH, HLS or both. Among the popular open-source media players, we mention:

• •

  The dash.js reference player [50] is an initiative of the DASH-IF to establish a production quality framework for building video and audio players that playback DASH content using client-side JavaScript libraries leveraging the Media Source Extensions (MSE) and Encrypted Media Extensions (EME) APIs defined by the W3C.

• •

  Shaka player [67] is a JavaScript library for HAS, developed by Google. It plays adaptive media formats (such as DASH and HLS) in modern browsers using the MSE and EME APIs.

• •

  hls.js player [55] is another JavaScript library developed by multiple companies. It relies on HTML5 media and MSE for playback. It works by transmuxing MPEG-2 Transport Stream (TS) and AAC/MP3 streams into ISO BMFF (MP4) fragments.

• •

  ExoPlayer [66] is a DASH, HLS and Microsoft Smooth Streaming (MSS) media player for Android, developed by Google. It provides an alternative to Android’s MediaPlayer API for playing audio and video locally and over the Internet.

Over the last few years, these media players have been extended to support LLL streaming protocols as described in Table V. Following the AWS recommendations [11], we conducted experiments over the AWS Elemental Media Services infrastructure [12] to investigate the achieved performance for these open-source media players with low-latency mode enabled (CMAF+CTE), i.e., dash.js (v3.2) [50], Shaka (v3.1) [67], hls.js (v1.0.0-rc.4) [55] and ExoPlayer (r2.13.2) [66], using four main scenarios. The main goals of these experiments are to show the following: ($i$) the impact of media player parameters on latency and quality, ($ii$) the performance of the E2E workflow architecture for LLL streaming over the Internet, and ($iii$) different deployment scenarios of LLL streaming over the AWS Elemental Media Services. Please note that these media players are continuously being developed and improved. Thus, our experimental results reflect only a snapshot in time. The main point is to illustrate that media players are often designed and tuned differently to achieve different, sometimes contradictory, performance trade-offs.

Fig. 20 shows four deployment scenarios for media distribution using the AWS Elemental Media Services. These scenarios are summarized in Table VI and aim to investigate the combination of different AWS Elemental Media Services, i.e., MediaLive, MediaPackage, MediaStore, Live and Delta, and AWS CloudFront CDN. Specifically, one workflow is fully deployed on-premises (Scenario 1) and three hybrid workflows (Scenarios 2–4) with encoder and packager or origin are deployed on-premises. These scenarios are similar to how Twitch uses AWS services to deliver content to its massive audience. For video contribution, a live feed at 1080p was created and transported from the live source to the encoder using RTMP.

These experiments can provide a general overview of how to decide and adjust an LLL streaming distribution workflow, including player parameters with the AWS Elemental Media Services, to meet the requirements of the target application. As the live source, we used the Big Buck Bunny (https://peach.blender.org/download/) sequence with a segment duration of five seconds and a chunk (or part) duration of one second (equal to 30 frames at 30 fps). Consistent with existing CMAF-based live services, we encoded the video with FFmpeg (H.264/MPEG-4 AVC codec) into four video representations {[email protected] Mbps, 480p@1 Mbps, 720p@2 Mbps and 1080p@5 Mbps} [11] and the audio into one audio representation at 128 Kbps. The session duration was fixed at 600 seconds. We used the Google Chrome browser (v86) with Fetch API support to run the browser-supported media players (i.e., dash.js, Shaka and hls.js), and Android (v10) to run ExoPlayer over WiFi home network conditions. The encoder, packager and origin server were in the USA, while the CDN edge and players were in Singapore. We fixed the target latency to five seconds, set the ABR scheme to heuristic hybrid-based (buffer + throughput), and we used LL-DASH or LL-HLS for the media distribution workflow, depending on the support in the player. For each experiment, we ran two tests—one with the default parameters (see Table V) and another with the well-tuned configurations of these parameters for LLL streaming—and re-visited the bandwidth measurement module [106] to avoid inaccuracies in measurements caused by CTE, as explained in Section VI-A.

Table VII shows the results achieved by each media player in the four scenarios in terms of viewer QoE metrics [106]: E2E latency, selected quality (bitrate), rebuffering, startup delay and playback speed. The key takeaways of these experiments are the following. (Note that these are results at the time of the experiments. Newer or older versions of the media players may deviate.)

• •

  Each player performed better with the well-tuned parameters than the default ones. Across all scenarios and player implementations, players with the well-tuned parameters achieved an average E2E latency of 5.11 seconds (near the target latency of five seconds), high quality, less rebuffering and low startup delay, compared to an E2E latency of 7.27 seconds for players with the default parameters. We note that players with the well-tuned parameters streamed at normal speed more than 85% of the time, with the remaining 15% either slower or faster during the whole live session. We expect this to have minimal impact on the visual quality.

• •

  The players (of all implementations) achieved better results in Scenario 4, followed by Scenario 1, compared to other scenarios. This is because of the deployment of components on-premises compared to the cloud. Moreover, AWS Elemental Live with AWS Elemental MediaStore for distribution shows a good fit for LLL streaming. In Scenarios 2 and 3, AWS Elemental MediaPackage and MediaLive add more delay to the latency because of repackaging and transcoding.

• •

  In all scenarios, dash.js achieves the best performance compared to the other open-source players, likely because dash.js is better tuned for LLL scenarios. We note that the other players are still working on improving their algorithms for LLL streaming.

• •

  For media distribution at last-mile delivery, a player’s network connection can significantly impact the latency. For example, a viewer may connect via a wired network at his/her home, via a WiFi access point, or using his/her mobile connection to access the content. Also, the proximity of the closest CDN server may impact the latency.

Overall, improvements in results can be attributed to the well-tuned player parameters, accurate bandwidth measurements, playback speed adjustments, and the location of delivery workflow components. We also tested with a chunk duration of 33 milliseconds and some adjustments at CloudFront, and we achieved better performance (not shown), which confirms that reducing latency requires an E2E workflow that is all streamlined and tuned together. We note that the media players are continuously being updated and their performance may vary across versions.

Finally, an E2E LLL streaming workflow can be further improved through analytics. One option is to use a centralized entity to collect data from each component in the delivery workflow by leveraging a Server and Network Assisted DASH (SAND) [84, 165] architecture or the Common Media Client Data (CMCD; CTA-5004) [44, 28] framework. Then, AI-based techniques can be used to make suitable decisions for better LLL delivery. For example, [28] demonstrated how the playback buffer status could be conveyed from the client to the server via CMCD to allow the server to dynamically allocate its outbound bandwidth according to each client’s buffer level (e.g., clients with critically low buffer levels are allocated a larger portion of the bandwidth). Similarly, the authors of [130] used three CMCD parameters, i.e., top bitrate, device type and screen resolution, to construct a suitable bitrate ladder for each device type (e.g., phone, TV or HDTV) at the server side. The companion framework, Common Media Server Data (CMSD; CTA-5006) [46, 107, 140, 29], is designed to allow data to be conveyed from the server to the client to enhance media delivery. This could also open up new possibilities and use cases, such as multi-CDN performance monitoring and switching at the client [132].

The grand challenge listed the following requirements based on Twitch’s experience with delivering LLL streams to users:

• •

  Small Playback Buffer Size: The media player should maintain a playback buffer slightly smaller than the target latency. When the target latency is one second, the player will have less than one second to adapt to network conditions to avoid rebuffering or deviations from the target latency.

• •

  Playback Speed Adjustments: The media player’s playback speed controller should adjust the playback speed appropriately to meet the target latency constraint while not risking rebuffering events.

• •

  Bandwidth Measurements: The media player should measure the available bandwidth accurately under varying network conditions. Furthermore, it should make a timely request for the segments to be delivered via CTE as soon as possible. In other words, the media player should send the request for the segment as early as possible as long as it does not arrive before the first chunk is available. This effectively removes the forward-trip time from the segment request and allows a more stable download rate once the subsequent chunks are available for streaming.

• •

  Robustness to Diverse Network Conditions: The media player should perform well across various network conditions. For example, the media player has to balance the number of bitrate switches against minimizing the number of rebuffering events while streaming at an acceptable quality and latency.

• •

  Fairness among Media Players: Multiple media players sharing the same bottleneck network may compete for network resources. For proper operation, media players should respect the fairness constraint (refer to Section VI-B).

In summary, contestants entering the challenge needed to design an ABR scheme tailored toward LLL streaming with CMAF and CTE in the one-to-two-second latency range. The ABR scheme was expected to minimize the number of rebuffering events while maximizing bandwidth utilization and respecting the other challenge requirements mentioned above.

In order to standardize and streamline development, Twitch provided a local, end-to-end testbed [167] to evaluate different implementations. The evaluation testbed is depicted in Fig. 21 and consists of three key components: the streaming engine, the network simulator and the metrics recorder, which are further detailed below. If the readers are interested in deploying the testbed and evaluating their customized ABR schemes, they may refer to the guidelines presented in [167].

• •

  Streaming Engine: This delivers and plays a live streaming session frame-by-frame using CTE. It is comprised of four modules as follows: ($i$) A live source that captures the 720p rendition of the Big Buck Bunny (BBB) sequence⁴⁴4[Online] Available: https://peach.blender.org/download. ($ii$) An FFmpeg encoder with H.264/AVC codec that encodes the BBB sequence into three representations {360p@200 Kbps, 480p@600 Kbps, 720p@1,000 Kbps}, with a segment duration of 0.5 seconds and a chunk duration of 33 ms (one frame duration at 30 fps). The encoded streams are then packaged in CMAF format. ($iii$) An origin and edge DASH server, which uses CTE to deliver chunks to the player. ($iv$) A dash.js reference player (v3.0.1) [50] that streams the live content.

• •

  Network Simulator: This throttles the bandwidth between the edge DASH server and the dash.js player according to predefined network profiles to mimic real-world scenarios. It is comprised of the following modules: ($i$) A Chrome dash.js player controlled by Puppeteer⁵⁵5Available [Online]: https://pptr.dev. ($ii$) A node.js⁶⁶6Available [Online]: https://nodejs.org application to run the tests with the respective network profiles. ($iii$) The Chrome DevTools protocol to throttle the bandwidth. ($iv$) Five different network profiles, namely {Cascade, Intra-Cascade, Spike, Slow-Jitters and Fast-Jitters} as shown in Table VIII. The evaluation testbed waits for the player to play the highest-bitrate representation for ten seconds before the simulation begins (for warm-up stabilization purposes).

• •

  Metrics Recorder: This listens to the dash.js player events and records metrics at each segment download, including selected bitrate, number of rebuffering events and their durations, number of bitrate switches, startup delay, E2E latency, bandwidth measurement and buffer level.

All competing submissions were evaluated on a workstation running Ubuntu 19.10 with two virtual machines (VMs). The first VM ran the dash.js player executing in Chrome v81. The second VM hosted the FFmpeg encoder with CMAF packaging and the Python-based origin and edge DASH servers. For network emulation, the Chrome DevTools was used to throttle the bandwidth between the edge server and player according to the provided network profiles. The target latency was fixed to one second.

The submissions were evaluated based on the five network profiles (results were averaged over five runs for all profiles) and the following criteria: average selected bitrate, average E2E latency, average number of bitrate switches, average playback speed and average rebuffering duration. These measurements were fed into the QoE model given in (2) [106] to produce a single score, which was used to rank the submissions. The QoE model is defined as follows:

The list of the notations for the QoE model is given in Table IX. The weight factor of each QoE criterion was fixed based on subjective tests [186, 185] as follows: $\alpha$ = segment duration (0.5 seconds), $\beta$ = maximum encoding bitrate (1,000 Kbps), $\sigma$ = minimum encoding bitrate (200 Kbps), $\mu$ = 1 second, and $\gamma$ = {if $L\leq 1.1$ seconds then $\gamma$ = 0.05 $\times$ minimum encoding bitrate, otherwise $\gamma$ = 0.1 $\times$ maximum encoding bitrate}. The playback speed was usually: ($i$) normal: 1$\times$, ($ii$) fast: e.g., 1.5$\times$, or ($iii$) slow: e.g., 0.95$\times$. Thus, if the playback speed were normal, then the playback speed penalty would be zero. Otherwise, the penalty was set to the minimum encoding bitrate level.

In total, three solutions were submitted to the challenge: LoL (Low-on-Latency) [106], L2A (Learn-to-Adapt) [94], and Stallion (STAndard Low-Latency vIdeo cONtrol) [73]. More details on these solutions (including their source codes) can be found in their original papers. For LoL, we present the results of the learning-based ABR scheme. As the baseline, the default ABR scheme of dash.js (referred to as Dynamic, which uses a hybrid of BOLA [159] and throughput-based ABR) was used.

The average results achieved over five runs and all network profiles are highlighted in Table X. Tables XI, XII, and XIII show the comparison results (average, median and standard deviation) for LoL vs. dash.js, L2A vs. dash.js and Stallion vs. dash.js, respectively, over five runs and all network profiles. Note that Stallion used a different target latency configuration (1.5 seconds instead of one second as used by all other solutions), contributing to its better rebuffering performance.

From the results, one key observation is that each solution has its own merits, such that it can achieve better results on specific QoE criteria or perform better under certain network conditions. For interested readers, the full details of the results can be found in [105].

Discussing more results, LoL achieved the highest average selected bitrate in all network profiles among the submissions due to its accurate bandwidth measurement (with an average accuracy of more than 96% across all network profiles). Moreover, LoL uses a learning-based ABR, which can track four dimensions with four different weights, including latency, bandwidth, buffer level and QoE, making the solution more sensitive to changes in any of these QoE criteria. Hence, it can quickly detect bandwidth changes and handle sudden bandwidth drops better. In contrast, dash.js, L2A and Stallion suffered from bandwidth overestimation or underestimation issues, resulting in lower bitrate selections, where inter-chunk idle times are more likely to occur. This follows our earlier discussion on how CTE may cause issues in existing bandwidth measurement techniques (refer to Fig. 19).

Twitch also observed that the performance of each solution varied under different network conditions. This suggested that different ABR solutions had their own strengths, and thus, using multiple ABR algorithms, combined with an appropriate ABR switching module that selects the best ABR scheme based on changing requirements and environmental conditions, may help us leverage these solutions better. One can use machine learning techniques for such an ABR switching module.

In addition to objective tests, Twitch also conducted subjective tests internally to evaluate the submissions. The subjective tests aimed to answer the following questions: Do you think this solution could perform well on the diverse Twitch network? What is the potential for each design? Have the authors realized this potential? Based on the responses, Twitch concluded the following:

According to Twitch, and considering the output of both objective and subjective evaluations, picking the winner was not easy as the three solutions were similar in overall performance. In the end, Twitch announced the winners as follows: (1) L2A, (2) LoL and (3) Stallion. We note that the authors of LoL have considered Twitch’s feedback and extended their solution in the follow-on LoL⁺ [25]. LoL⁺ avoids the hard-coded weights of its learning-based rule by implementing a heuristic-based algorithm to adjust the weights dynamically. It formulates the weight selection as an optimization weight assignment problem and solves it using a heuristic-based algorithm running in two states. ($i$) At the beginning of a live session (buffering-filling state), the weight selection algorithm uses a k-means++ mechanism [19] to find suitable initial weights for the considered features, i.e., individual QoE metrics in (2) and bandwidth measurement. ($ii$) After the buffering-filling state (steady state), it uses a weight assignment algorithm that dynamically adjusts these weights at every segment download, allowing better performance. LoL⁺ also includes a learning-based bitrate selection rule decoupled from the QoE function and selects based on individual QoE criteria in (2) instead. Finally, LoL⁺ adopts a hybrid proactive approach to control the playback speed, jointly considering the current latency and current buffer level. Therefore, it aims to eliminate issues related to latency and buffer occupancy (e.g., latency increase, rebuffering events and quality drops) before they occur. The detailed results and discussion of LoL⁺ against LoL, L2A and Stallion are available in [25].