Virtual Reality: A Survey of Enabling Technologies and its Applications in IoT

As a VR user can change viewport position by moving his or her head, head movement traces can be used to study a user’s head motion pattern. The general tracing collection procedure is shown as follows.

• •

  Initially, when watching a VR video streamed to the HMD, viewers can look around to change their viewport position freely and get familar with the HMD.

• •

  During the video playback, viewers’ head movement data need to be recorded. [28] used OpenTrack as the sensor logger to record the viewer orientation, while [29] developed a software to log the viewer’s head position at each frame.

• •

  After recording, the head motion data is stored in .csv, .txt or some other format.

There have been many studies on the characteristics and statistics of the head movement traces. For example, [30] visualized the paths of their viewers’ gazing directions in 3D space, and analyzed the angular velocity of their head movement. [31] analyzed fixation count per latitude and per longitude to provide some insights about 360-degree content exploration. On top of that, some researchers also considered measuring viewing experience. [34] asked the viewers to fill in the simulator sickness questionnaire and analyzed the result.

Eye movement traces are essential to study users’ gaze behaviors [36]. However, compared to head movement traces, it is more difficult to collect eye movement traces, since some auxiliary equipment (e.g., eye-tracking camera) is usually needed. [31] used an HTC VIVE headset equipped with an SMI (SensoMotoric Instrument) eye-tracker to display VR streaming and capture eye movement data. [37] captured viewers’ eye fixations with an HTC VIVE headset and a 7invensun a-Glass eye tracker (an in-helmet camera system). On top of that, some studies were conducted based on their eye tracking dataset, including consistency of gaze points, distribution of eye fixations in latitude and longitude, etc. Besides, some researchers use self-developed testbeds to replace the eye-tracking equipment. [32] designed a testbed to gather viewer’s viewport center trajectories when watching VR streaming without the need of specific eye tracking devices.

As the content of VR videos and viewer’s unique viewing habit affect a viewer’s future viewport jointly, it is necessary to take content traces into consideration. [28] generated image saliency maps and motion maps for each 360-degree video they collected. With content traces, content-based algorithm can be applied to predict future viewport. [38] proposed a fixation prediction network that concurrently leverages content and sensor traces.

In summary, Table II compares the characteristics of the three types of datasets. Table III specifies the main properties of the 360-degree videos used by the datasets.

The user driven solution indicates that the past user head movement traces are used in predicting the future FoV. Based on the utilized prediction methods, it can be further split into two kinds of models.

• •

  Linear regression (LR). [39] used past head motion to predict future head motion based on linear regression [45]. Except LR, [40] also used ridge regression (RR) and support vector regression (SVR) with different prediction windows to predict future FoV.

• •

  Long short-term memory (LSTM). [20] took the viewpoint matrix in past two seconds as input of the LSTM model and predicted the probabilities over FoV. [41] compared the LSTM model for FoV prediction with LR algorithm, image-based CNN model [38] and KNN algorithm [44]. The comparison result showed that the LSTM model performs the best.

The characteristics of the content can also help predict FoV. Intuitively, there are also two solutions for the content driven FoV prediction.

• •

  Convolutional Neural Network (CNN). This method is to predict the viewpoint from the saliency map inferred from a CNN model. In other words, the CNN model is used to generate the saliency map from the video frame, and there is a known mapping relation between the saliency map and the FoV. For example, [43] proposed a tile-based approach using a saliency map that integrated the information of human visual attention with the contents.

• •

  LSTM. However, it requires much computation power on inferring the saliency map from each frame, which will also incur the playback delay. As a candidate, the LSTM model can take the past saliency maps as the input and predict the future saliency map. The LSTM method is preferred in the current studies. For instance, [42] integrated the panoramic saliency maps with user head orientation history for head movement prediction. It took saliency map and head orientation map as input and outputs the predicted head orientation of the next moment. [38] concurrently leveraged sensor-related and content-related features to predict the viewer fixation in the future. The sensor-related features include HMD orientations, while the content-related features include image saliency maps and motion maps.

Combining the user driven method and the content driven method, some work intended to design a user-and-content driven FoV prediction framework. [44] designed such a work, which integrated past fixations with cross-users’ region-of-interest (ROI) found from their historical fixations. Benefiting from data-driven learning, they proposed a cross-user learning based system (CLS) to improve the precision of viewport prediction. Since users have similar ROI when watching a same video, it is possible to exploit cross-users’ ROI behavior to predict viewport. The authors used the DBSCAN clustering algorithm [46] to group together the users’ fixation that are closely packed together, i.e. fixations that have enough nearby neighbors are grouped together as a class. The viewing probability can be predicted by class.

The comparison of different FoV prediction methods is summarized in Table IV.

For 360-degree videos, the contents are generally built on 2D frames. The 2D frames are first encoded at the content transmitter and then decoded for efficient video transmission. Afterwards, the client can project and further render the decoded 2D frames into 3D space.

A point cloud is a set of data points in space. The points represent a 3D shape or object. Each point has its set of X, Y and Z coordinates. Different from conventional 2D frame structure, the volumetric video represented with 3D point cloud format enables users to watch the video with a highly immersive and interactive user experience [47].

3D point cloud enables the ability to transmit and interact with holographic data from remote locations across a network. Holographic type communications (HTC) allows remote participants to be projected as holograms into a meeting room to facilitate real-time interactions with local participants. Likewise, training and education applications can provide users the ability to dynamically interact with ultra-realistic holographic objects for teaching purposes [48]. However, HTC videos need a type of spectrum band that is currently unavailable in the millimeter-wave spectrum (5G). This situation presents significant challenges related to area or spatial spectral efficiency and the needed frequency spectrum bands for connectivity [49]. Hence, a bigger radio frequency spectrum bandwidth has become necessary and can only be found at the sub-THz and THz bands, often referred to as the gap band between the microwave and optical spectra [50, 51].

Different from standard video encoding, it is the tiling method on the projected map that should be specially considered for the VR streaming. Accordingly, the encoding scheme can be categorized into non-tiling, uniform tiling and non-uniform tiling schemes. The comparison of different encoding schemes is shown in Table V.

The non-tiling scheme encodes the VR streaming as traditional video streaming. It streams each frame as a whole to the HMD. [52] adopted the non-tiling scheme that streams the VR video as a single-tile video. Videos are split into segments temporally as in standard DASH implementations and the bitrate adaption is in the segment level. The videos are encoded by [53] with the [54] encoder.

As only a small portion of each video frame displayed on the screen at a specific time, streaming the entire frame with the same quality is a waste of bandwidth. Many methods adopted uniform tiling schemes due to the users’ dynamic viewports [55, 7, 56]. To support viewport-adaptive streaming, video segments are further spatially split into equal-sized tiles. Different tiles can be allocated with different bitrate levels. Spatial representation description (SRD) has been proposed as an improvement to the MPEG-DASH standard [57]. SRD can be used to describe the spatial relationship among tiles that belong to the same temporal video segment within the DASH media presentation description file. This update in the MPEG standard promotes the deployment of tile-based streaming systems.

According to [3, 58, 59, 60], fixed-size tiling approaches suffer from reduced encoding efficiency. OpTile [3] is a scheme that estimates per-tile storage costs and then solves an integer linear program to obtain optimal non-uniform tiling. ClusTile [58] enhanced OpTile by clustering the collected user views. MiniView [59] encoded the 360-degree videos in the MiniView Layout which uses smaller-FoV projections for the purpose of saving the encoded video size while delivering similar visual qualities. Similarly, considering the perceived quality, [60] proposed a variable-sized tiling scheme.

In VR streaming, a tile could refer to data of another tile in a previous or future reference frames, leading to decoding glitches. Therefore, [61] proposed to constrain the tiles encoding so that each tile only refers to the same tiles in previous or future frames. This reduces the complexity on the clients and in the networks. Following this concept, [28] also restricted the motion prediction when encoding the video tiles. Similarly, [62] designed a dynamic encoding technique to dynamically determine the RoI in order to reduce the transmission latency and bandwidth consumption in the offloading pipeline.

Some work focused on the volumetric video compression represented with 3D point cloud. For example, Google Draco ²²2Google Draco, https://google.github.io/draco/. and Point Cloud Library (PCL) ³³3Point Cloud Library, https://github.com/PointCloudLibrary/pcl., widely used open-source libraries to compress volumetric videos, decrease the data size by 4 times.

The widely used coding standards for 360-degree video are [63], [64] and the extended form scalable video coding (SVC) [65], which will be introduced as follows.

As shown in Figure 3(a), an equirectangular projection is something like the world map. The 360-degree view can be represented in a single aspect ratio image, with the expense of having distortion towards the poles. One approach to create an equirectangular projection is to first render a sphere map, and then use a tool like [68] to convert the distortion.

As shown in Figure 3(b), cubic is a type of projection for mapping a portion of the surface of a sphere (or the whole sphere) to flat images. The images are arranged like the faces of a cube, and this cube is viewed from its center. Four cube faces cover front, right, back and left, one the zenith and one the nadir, each of them having $90^{\circ}\times 90^{\circ}$ FoV. In each cube face, all straight lines stay straight, hence it is quite suitable for editing. To address the projection inefficiency of cubic projections, [70] recently proposed equi-angular cube (EAC). EAC attempts to address the problem of pixel inefficiency in cube faces by distorting the standard cubic projection to reduce projection inefficiency.

As shown in Figure 3(c), in the pyramid projection, the spherical video is projected onto a pyramid layout from central points to generate a set of video representations. The base of pyramid is in the viewing direction with full resolution, while the rest pixels are projected onto the sides of pyramid with decreasing resolution. However, the pyramid projection is especially sensitive to head movements, because sides of a pyramid corresponding to non-viewing directions are rendered with low resolution. Besides, since there are multiple viewports rendered and stored for the video, pyramid projection introduces extra overheads to a certain extent.

As shown in Figure 3(d), a rhombic projection can be dissected with its center into 4 trigonal trapezohedra [71]. These rhombohedra are the cells of a trigonal trapezohedral honeycomb. This is analogous to the dissection of a regular hexagon dissected into rhombi, and tiled in the plane as a rhombille.

Recently, [59] proposed the MiniView layout, which improves bandwidth efficiency of VR streaming by reducing the quantity of over-represented pixels in the spherical-to-2D projection. As shown in Figure 3(e), MiniView encodes a small portion of the sphere by applying the rectilinear projection with a small FoV.

The heuristic bitrate adaption schemes take maximizing bandwidth utilization as an objective and allocate the highest quality to the tiles in viewport [72, 73, 74]. [73] divided the tiles into different groups. With the constrained bandwidth, the tiles of viewport were streamed at a higher quality and the others were streamed at a lower quality. [72] explored various options about full delivery and partial delivery to enable the bandwidth efficient adaptive streaming of omnidirectional video over HTTP.

This category solves the bitrate adaption problem as an optimization problem. Rubiks [39] used an MPC-based optimization framework to optimize the user QoE in chunk level. CLS [44] constructed a Multiple-Choice Knapsack model to determine the rates for tiles. Pano [60] determined the bitrate of each chunk and maximized the overall perceived quality with total size constraint based on MPC. Flare [40] determined the quality level for each to-be-fetched tile by jointly considering different QoE objectives.

The deep reinforcement learning achieves great success in traditional video streaming. It also performs prominent at VR streaming in this two years [75, 41, 76, 77, 78]. These schemes directly optimized the use QoE and they all adopt the A3C to train the model. [75] used A3C-based model to determine the bitrates for both viewport and non-viewport areas. DRL360 [41] adaptively allocated rates for the tiles of the future video frames based on the observations collected by client video players. [76] determined the quality level of the tiles in estimated FoV. [77] determined the bitrates for multiple estimated FoVs. SR360 [78] adaptively determined the tile bitrate inside the viewport and enhanced the tile quality by super-resolution (SR) at client side.

Following [80, 81, 82, 83], we divide the playback delay into two key components, i.e., the end-to-end delay and the rebuffering delay, which will be introduced as follows.

In VR systems, the end-to-end delay mainly includes processing delay, transmission delay, propagation delay, queueing delay, and other kinds of delay. The propagation delay is the time for a packet to reach its destination, e.g., HMD. The rebuffering delay is the additional time needed for packet retransmission. These five key kinds of network delays are introduced as follows.

The processing delay for VR streaming transmission can be represented by coding/decoding delay. Current streaming strategies approaches found that traditional decoding and existing tile-based decoding cannot support 8K or higher video resolution. Correspondingly, [39] suggested that we do not have the luxury to allocate a tile to each decoding thread, but should have a different task-to-thread assignment.

The transmission delay is the time needed to transmit an entire packet, from the first bit to the last bit. The transmission delay is determined primarily by the number of bits and the capacity of the transmitter device.

The propagation delay is the time to propagate a bit through the communication link. It can be computed as the ratio between the link length and the propagation speed over the specific medium. Generally, the propagation delay is equal to $d/v$ where $d$ is the distance between the transmitter and the receiver and $v$ is the wave propagation speed.

The queueing delay is defined as the waiting time of packets in the buffer until they can be executed. Queueing delay may be caused by delays at the originating switch, intermediate switches, or the receiver servicing switch.

There are other sources of delay: i). Packetization Delay. Packetization delay is the time taken to fill a packet payload with encoded data. Packetization delay can also be called accumulation delay, as the data samples accumulate in a buffer before they are released. ii). Serialization Delay. Serialization delay is the fixed delay required to clock a data frame onto the network interface. It is directly related to the clock rate on the trunk. At low clock speeds and small frame sizes, the extra flag needed to separate frames is significant.

The playback and downloading strategy should adapt to the buffer occupancy. If the buffer occupancy reaches 0 (i.e., no chunk is left in the buffer), the playback will suffer from rebuffering [41]. The playback will continue until there is at least one chunk in the buffer. Let $b_{k}$ denote the buffer occupancy when the $k$-th chunk has been downloaded completely. Therefore, after the startup stage, the buffer occupancy changes according to the downloading of the new chunk and consumption of chunks in the buffer, i.e.,

where $T$ is the length of a video chunk, $R_{k}$ is the sum bitrates for caching chunk $k$, and $C_{k}$ is the downlink rate for chunk $k$ that can be estimated [41].

Following [5], we prefer to let the buffer occupancy stay at least $B_{\text{target}}$, that is $b_{k}\geq B_{\text{target}}$. Then the sum bitrates for caching video chunk $k$ should satisfy:

Generally, we set a lower bound $R_{\min}$ to $R_{k}$, then we have:

The term “video quality” is defined in terms of fidelity, i.e., how closely a processed or delivered signal matches the original source (or reference) signal. The main concern is to detect and quantify any distortions that have been introduced into the signal as it passes through a network or device. The video quality metric can be further split into three parts, i.e., subjective score, objective score, and quality variance.

Subjective testing is the only proven way to evaluate video quality [84]. For example, human subjects are asked to assess the overall quality of the processed sequence with respect to the original (reference) sequence. Unfortunately, this mode of testing is very expensive, time-consuming, and often impractical. The subjects score the processed video sequence on a scale corresponding to their mental measure of the quality, termed Mean Opinion Score (MOS). As shown in Table VII, when the MOS score is on a 1 to 5 scale, corresponding opinions for these scores are Unacceptable, Poo, Fair, Good and Excellent, respectively. Note that the MOS values are different for various stalling events under different bitrate levels [85]. [86] showed that video stall severely impacts the QoE, much stronger than other typical impairments like initial delay or quality switching. Furthermore, high-quality 360 videos at 4K resolutions can provide good perceptual quality score (around 4.5 MOS) to users [87].

Specifically, as one of the key components of MOS, cybersickness is the feeling of nausea and dizziness when watching virtual reality video. [88] subjectively evaluated the impact of three QoE-affecting factors (gender, user’s interest, and user’s familiarity with VR) on cybersickness. The statistical results from the subjective experiment provided various findings on QoE impact factors in terms of perceptual quality and cybersickness. To predict the QoE, a neural network model was trained and tested on cybersickness data obtained from the subjective test.

As the subjects are difficult to obtain, a number of algorithms have been developed to estimate video quality by using mathematical analysis in place of human observers. The widely used metrics include sum bitrate, quality-rate (Q-R) function, peak signal-to-noise ratio (PSNR), and structural similarity (SSIM).

In Table VIII, we can observe that the bitrate metric is almost used in most literature, except for work focusing on subjective scores (e.g., [93, 94]). The sum bitrate is the metric that is most focused in the schedule of VR streaming.

In the VR streaming problem formulation, the bitrate metric, as the target scheduling parameter, can be regarded as the constraint or the objective. In [89], the scheduling constraint is the sum of the byte-stream sizes for every area is equal to $B$, where $B$ denotes the transmission bandwidth. Generally, the sum bitrates allocated to tiles is less than the transmission bandwidth [75]. In [39], the objective of the VR streaming design is to maximize the bandwidth saving, i.e., minimize the bandwidth cost.

Instead of directly using the sum bitrate metric, some work defined the VR streaming quality as a mapping function from the bitrate. For example, according to [95, 22], user QoE has a logarithmic relation following the rate distortion theory, i.e.,

where $r_{i}^{(k)}$ represents the allocated bitrate for tile $i$ of chunk $k$ and $\eta$ is a constant obtained from field measurements. In total, we have the QoE on video quality for all tiles as:

Peak signal-to-noise ratio, often abbreviated PSNR, is an engineering term for the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. Because many signals have a very wide dynamic range, PSNR is usually expressed in terms of the logarithmic decibel scale. PSNR is most easily defined via the mean squared error (MSE). Given a noise-free $m\times n$ monochrome image $I$ and its noisy approximation $K$, MSE is defined as:

The PSNR (in dB) is defined as:

where $\text{MAX}_{I}$ is the maximum possible pixel value of the image. When the pixels are represented using 8 bits per sample, this is 255. More generally, when samples are represented using linear PCM with $B$ bits per sample, $\text{MAX}_{I}$ is $2^{B}-1$. A PSNR value of 35dB is generally considered good.

Recently, [96] proposed an improved PSNR metric, named viewport PSNR (V-PSNR), which can directly indicate the quality of content in the user’s viewport.

The SSIM index is a method for predicting the perceived quality of all kinds of digital images and videos. The SSIM index is calculated on various windows of an image. The measure between two windows $x$ and $y$ of common size $N\times N$ can be calculated as:

where $\mu_{x}$ and $\sigma_{x}^{2}$ are the average and the variance of $x$, and $\sigma_{xy}$ is the covariance of $x$ and $y$. Two variables $c_{1}=(k_{1}L)^{2}$ and $c_{2}=(k_{2}L)^{2}$ are set to stabilize the division with weak denominator, where $L$ is the dynamic range of the pixel-values and $k_{1}=0.01$, $k_{2}=0.03$ following [97].

The SSIM formula is based on three comparison measurements between the samples of $x$ and $y$: luminance, contrast and structure. The SSIM actually measures the perceptual difference between two similar images. It cannot judge which of the two is better: that must be inferred from knowing which is the “original” and which has been subjected to additional processing such as data compression. Unlike PSNR, SSIM is based on visible structures in the image.

However, the SSIM index achieves the best performance when applied at an appropriate scale (i.e. viewer distance or screen height). Calibrating the parameters, such as viewing distance and picture resolution, create the most challenges of this approach. To rectify this, multi-scale, structure similarity (MS-SSIM) has also been defined [97]. In MS-SSIM, the picture is evaluated at various resolutions and the result is an average of these calibrated steps. It has been shown that MS-SSIM outperforms simple SSIM even when the SSIM is correctly calibrated to the environment and dataset.

If the quality of the content is not smooth, it will decrease the users’ QoE, and we calculate this value according to the coefficient of variation of quality of content in the viewport. It is also essential to consider the measure of quality switching (i.e., the temporal variation) between consecutive chunks. The rate changes between two consecutive chunks may impulse users with physiological symptoms such as dizziness and headache. Hence, quality switch should not be drastic, which can be measured by:

where $r_{i}^{(k)}$ denotes rate of tile $i$ at the $k$-th chunk.

Peak signal-to-perceptible-noise ratio (PSPNR) is an extension of PSNR [98]. It improves the classic PSNR by filtering out quality distortions that are imperceptible by users. PSPNR is proposed to assess the quality of the compressed image by taking the perceptible part of the distortion into account. The key to PSPNR is notion of just-noticeable-difference (JND) [99], which is defined by the minimal changes in pixel values that can be perceived by viewers. The perceptible distortion of the compressed image is referred to the part by which the distortion is noticeable, or the part of distortion that exceeds the threshold given by the JND profile.

The current VR market lacks of UHD video sources, owing to the harsh demand for UHD 360-degree video acquisition equipment and the long video shooting process. Except for a low-cost design on video shooting devices, the VR content can also be generated from aggression of crowdsoured image or video clips [120, 121, 122]. However, such methods often have resource constraints in terms of bandwidth, storage, and processing capability, which limit the number of videos that can be crowdsourced. It is challenging to use limited resources to crowdsource video that best recover the 360-degree target area.

POI360, as the first POrtable Interactive 360-degree video telephony system, confronts an intrinsic conflict between the strict latency requirement of interactive video and the highly dynamic cellular network condition [102]. The situation is even more challenging when the heavy 360-degree video traffic pushes the cellular link to its limit, as a fluctuating link bandwidth can easily fall below the high traffic load. POI360’s rate control must detect this and reduce the traffic rate promptly to prevent video freezes.

Current wearable-device-based VR solutions are costly and inconvenient to carry. VR HMD has a built-in processor and battery, leading to a heavy HMD to wear. Therefore, current HMDs cannot offer us a lightweight yet high-quality VR experience. For most IoT applications, it is urgently required a lightweight VR solution design.

The current application-based VR solutions require installing the specific app in advance and lack cross-platform support (i.e., a VR activity of one IoT application cannot be used in other applications directly). [123] envisioned that web-based solutions with natural cross-platform advantages is attracting more and more attention and is opening up a new research field of VR system design.

Despite the above mentioned bitrate adaption design, we presents some potential means to improve the user QoE, i.e.,

• •

  Reducing the video decoding operation complexity, and thus the processing delay can be reduced.

• •

  Minimizing the transmitted content size (# bits), and thus the transmission delay can be reduced.

• •

  Maximizing the transmission bandwidth (bits per second), and thus the transmission delay can be reduced.

• •

  Enhancing the viewpoint prediction accuracy, and thus the rebuffering delay can be reduced. Meanwhile, this can also enhance the video quality in the viewport.

Among various research efforts towards the VR streaming, head movement prediction is one essential yet daunting task that needs to be addressed urgently. Besides enabling bandwidth-efficient VR streaming, predicting head movement accurately can significantly reduce the motion-to-photon delay since it would be possible to render exactly one viewport from the downloaded video portions in advance before the head moves. More importantly, the content view characteristics are various across different users. It is urgently required to design a personalized FoV prediction model to further improve the VR streaming efficiency.

Here we illustrate some extension ideas for find-grained FoV prediction, including

• •

  Adding external information. Besides the 3DoF or 6DoF information, there also exist external information that can be obtained from the sensors on HMD. For example, eye-trackers are installed in some recent HMD, where the eye-tracking information can highly likely to improve the FoV prediction performance.

• •

  Exploiting accurate user portrait. Current FoV prediction schemes are based on user portrait from historical view trajectory, which might less accurate.

• •

  Brain electrical information might also help improve the FoV prediction accuracy.

Compared to the traditional device-to-device communication, the connection between HMDs is also urgently required but faces much more challenges. First, the VR contents are usually UHD with high data volume, thus the radio technology should guarantee a high downlink rate. Moreover, the VR users are always with dynamic movement, thus the signals with new radios (e.g., mmWave or VLC) might be easily blocked and introduce an obvious decline on the received signal strength.

The VR systems generate, collect, and analyze large volume of data to derive intelligence, privacy becomes a great concern. The privacy issue is caused by several special characteristics of both VR technologies and IoT application requirements. The collection of user dynamic view behaviors and characteristics may pose threats to the privacy of people’s preference information. In VR systems, the multiple-dimensional information leakage might pose users under a more dangerous situation, which will be a matter of continuing concern.

Although many communities have contributed some great efforts on public dataset generation as discussed in Section III, it is still not enough for efficiently mining key characteristics from the limited amounts of user traces. We need more contribution on public datasets, with which the needs of the research community on VR streaming can be efficiently met.