User-centric Heterogeneous-action Deep Reinforcement Learning for Virtual Reality in the Metaverse over Wireless Networks

The recent futuristic notion of Metaverse is an extension, a complete simulation, and a mirror of the real world, which is empowered by the maturing high-performance Extended Reality (XR) technologies. Among them, Virtual Reality (VR) provides users with a fully immersive and digital world, which has been used in many fields such as entertainment, socialization, and industry [2, 3]. To alleviate the obtrusive sense of restriction for users when moving around wearing VR devices, the mobility of these devices is essential. Feasible solutions are applied by using wireless connections and decreasing the weights at the sacrifice of local computation abilities. Thus, even state-of-art VR devices (e.g., HTC Vive [4]) do not have the sufficient local computing power to support high-resolution and high-frame-rate applications. In this event, transferring some VR frame generations to a remote server is necessary. Furthermore, under the context of the Metaverse, the boundary between virtual and physical environments becomes more and more blurred, which brings more frequent demands of accessing the virtual world from users. As the Metaverse emphasizes building the bridge between the virtual and real worlds and supporting user inputs anywhere and anytime, it is necessary to consider how to allocate the limited resources to a wide variety of VR users (VUs) with distinct characteristics and demands.

We first explain the challenges with motivations for this work from the following aspects.

User-centric features and user-diverse problems. The main reason for its popularity can be contributed to the user-centric services, which serve as the fundamental basis of the next-generation Internet [2]. The Metaverse is a perceived virtual universe and ideally, allows people to access it anywhere and anytime, with any purpose. Compared to the traditional network structure, the user-centric features also pose a user-diverse problem: the wide variety of use cases (applications) and user-inherent characteristics (e.g., device types, battery power) induce highly different demands for frames per second (FPS), resolutions, etc. Further, the much larger size of transmitted data in VR can pose a challenge for devices with lower capabilities. These devices may struggle to complete tasks without sufficient support from remote computing resources. Therefore, the first and foremost challenge is how to achieve efficient utilization of those network resources in the multi-user scenario where each user has a different purpose of use and requirements. This propels us to seek a more user-centric and oriented solution to handle widely different users.

Wireless communication for VR. VR is a key feature of an immersive Metaverse socialization experience. Compared to traditional two-dimensional images, generating $360^{\circ}$ panoramic images for the VR experience is computationally intensive. However, as the mobility of VR devices is attached with great importance under the Metaverse context, manufacturers have to lessen the weight at the cost of local computation capability. As a consequence, the existing VR devices lack the local computation ability of high-resolution and frame-rate applications. A feasible solution to powering an immersive socialization experience on VR devices is to make the Metaverse Server help the application frame generation, and send generated frames to VUs. However, with the frequent demand and high congestion degrees for network resources, it is necessary to lighten the network burden by allocating some devices with higher local computing power to do local generation sometimes. Therefore, this paper considers two cases: (i) server generation, and (ii) local generation.

Joint optimization in wireless communication. In many scenarios, there are more than one important objectives that need to be optimized. In the wireless scenario where two cases of server and local generation are taken into account, two parts are detrimental to data transfer efficiency: (i) The channel access arrangement for VUs (including being assigned with no channel and doing local generation), and (ii) transmission power allocation for VUs. Based on this, this paper focuses on jointly optimizing these variables in the wireless downlink transmission scenario, and considering the VUs’ diverse characteristics.

The related work is separated into multiple aspects according to the challenges and motivations since we design our work and make contributions from these multiple domains. The references and our novelties compared to them are expounded in the following.

User-centric Metaverse. For the time being, people are already very aware of the user-centric particularity in the Metaverse. Lee et al. [2] claim that the Metaverse is user-centric by design, and it will rely on pervasive network access. Consequently, users with diverse purposes, devices, and demands can access the universe anytime and anywhere. Du et al. [5] emphasized the user-centric demands in the Metaverse and proposed an attention-aware network resource allocation considering the diversities among users’ interests and applications. Both papers above came up with attractive and fresh concepts and discussed the potential challenges and future directions. However, none of them has ever studied a specific scenario with problems and designed novel algorithms to tackle them. In this paper, we design a concrete user-diverse problem scenario and invent a user-centric DRL structure correspondingly.

VR over wireless communication. In recent years, VR services over wireless communication have been thoroughly studied in many previous works. Yang et al. [6] investigated the problem of providing ultra-reliable and power-efficient VR strategies for wireless mobile VUs by Deep Reinforcement Learning (DRL) algorithms. In order to fit the discrete action space required in their DRL algorithm, they quantize the continuous actions into discrete actions. Xiao et al. [7] studied the predictive VR video delivering by optimizing the bitrate with DRL methods. Some other works has also demonstrated the excellent performance of DRL methods over wireless communications as its ability to explore and exploit in self-defined environments [8, 9, 10]. However, none of the previous works considered the varying purpose of use and requirements between different VUs, and there are no existing works that have designed a user-centric and user-oriented DRL method as our proposed solutions.

Joint optimization in wireless communications with DRL methods. Some remarkable works have investigated using DRL methods to solve joint optimization problems [11, 12]. For instance, Guo et al. [13] solved the handover control and power allocation joint problem using Multi-agent Proximal Policy Optimization (MAPPO) and obtained satisfactory results. As mixed continuous-discrete actions can lead to problems of directly embedding the MAPPO algorithm, they use discrete power requirement factors instead of a continuous power allocation to simplify the problem. Thus, the problem they aimed to address does not have heterogeneous actions (both discrete and continuous actions), hence they can directly use MAPPO. He et al. [14] researched the joint optimization problems on channel access and power resource arrangements by using DRL methods to determine the channel assignment and conducting traditional optimization methods for power allocation under the channel information. However, none of these works considers a user-diverse scenario or problems with heterogeneous actions. Compared to them, we propose a novel Multi-Agent Deep Reinforcement Learning (MADRL) structure, which is equipped with a user-centric view and able to handle interactive and heterogeneous actions.

This paper proposes a novel multi-user VR model in a downlink Non-Orthogonal Multiple Access (NOMA) system. Specifically, we optimize the channel access arrangement and downlink power allocation jointly, taking the diversities between VUs into consideration. We designed a novel MADRL algorithm, User-centric Critic with Heterogenous Actors (UCHA), which considers the varying purpose of use and requirements of the users and handles the heterogeneous action spaces. In terms of the backbone of our algorithm, we re-design the widely-used Proximal Policy Optimization (PPO) algorithm [15] with a reward decomposition structure in Critic, and two assymetric Actors.

Our contributions are as follows:

• •

  Formulating user-centric VR in the Metaverse: We study the user-centric Metaverse over the wireless network, designing a multi-user VR scenario where a Metaverse Server assists users in generating reality-assisted virtual environments.

• •

  Heterogeneous Actors for inseparable optimization variables: We created two asymmetric Actors interacting with each other to handle the inseparable and discrete-continuous mixed optimization objectives. Specifically, Actor one is for the channel access arrangement, and Actor two is for the power allocation based on the solution by Actor one.

• •

  User-centric Critic for user-diverse scenario: We crafted a novel user-centric Critic equipped with a more user-specific architecture, in which we decompose the reward into multiple VUs and evaluate the value for each VU. To the best of our knowledge, we are the first to embed the hybrid reward architecture in multi-agent reinforcement learning, and use it to solve communication problems.

• •

  Novel and comprehensive simulation: We conduct comprehensive experiments and design novel metrics to evaluate our proposed solution. The experimental results indicate that UCHA achieves the fastest convergence speed and attains the highest rewards among all baseline models. UCHA’s allocations are more user-specific to individual users and appear to be more reasonable, as they fulfill different requirements of the users.

The rest of the paper is organized as follows. Section II introduces our system model. Then Sections III and IV propose our deep reinforcement learning setting and algorithm. In Section V, extensive experiments are performed, and various methods are compared to show the prowess of our strategy. Section VI concludes the paper.

A 6-page short version is accepted by the 2023 IEEE International Conference on Communications (ICC) [1]. In that conference version, we compare the user-centric structure in Critic with the normal Critic structure, and demonstrate its remarkable performance and ability to accelerates the convergence speed. Besides, we also demonstrate that using this structure in PPO is much superior to Hybrid Reward DQN [16]. However, in that version, we do not consider anything about the resolution, nor the multi-agent structure for further optimizing the transmission power, and these are all highlights of this journal version. Furthermore, this paper also designs new algorithms, metrics, and evaluation methods, compared to the conference version.

In terms of channel allocation, we define an $N\times T$ matrix $\boldsymbol{Z}$ such that the element in its $n$th row and $t$th column is $z_{n}^{t}$, for $n\in\{1,2,\ldots,N\}$ and $t\in\{1,2,\ldots,T\}$, indicating that the channel allocation for VU $n$ at $t$ is $z_{n}^{t}$. In other words, $\boldsymbol{Z}$ denotes the selection of downlink channel arrangement. Specifically, in the studied system of one Metaverse server and $N$ VUs, our channel allocation for each VU $n\in\{1,2\ldots,N\}$ includes two cases:

• •

  Case 1: Server-generated frame. If VU $n$ is assigned for a channel $m$ at time step $t$ (i.e., $z_{n}^{t}=m,m\in\mathcal{M}$), the Metaverse server selects the $m$th channel for downlink communication with VU $n$ to deliver the frame that the server generates for VU $n$. In this case, if the sum delay (for generation and transmission) of each frame exceeds slot duration $\iota$, this frame is deemed as a failure.

• •

  Case 2: VU-generated frame. If VU $n$ is assigned with no channel (i.e.,$z_{n}^{t}=0$), it generates the frame locally with a lower computation capability, and at the expense of energy consumption without communicating with the server. In this case, each VU assigned to do local generation will generate the frame of the highest in-time-processable resolution (i.e., can be generated in $\iota$). If the generated resolution is below the minimum acceptable resolution, this frame is deemed as a failure.

Thus, the channel allocation in this paper also includes the decisions on whether the frames for the VUs are generated by the server or the VUs (for simplicity, we sometimes just say channel allocation without mentioning decisions of frame-generation locations since the former includes the latter). In other words, $z_{n}^{t}=m,m\in\mathcal{M}$ indicates VU $n$ is arranged to channel $m$ at step $t$, and $z_{n}^{t}=0$ means user $n$ needs to generate locally. Next, we explain the two cases.

In each time step, the server will manage the downlink channels $\mathcal{M}$ of all VUs $\mathcal{N}$, and it will subsequently allocate the downlink transmission powers $\boldsymbol{p}^{t}$ with this Channel State Information (CSI). Here we define $\boldsymbol{P}:=[\boldsymbol{p}^{1},\boldsymbol{p}^{2},\ldots,\boldsymbol{p}^{T}]$ as the downlink power, where $\boldsymbol{p}^{t}:=[p_{1}^{t},p_{2}^{t},\ldots,p_{N}^{t}]$, $p_{n}^{t}$ is the power for VU $n$ at $t$, and enforce $\sum_{n\in\mathcal{N}}p_{n}^{t}\leq p_{max}$. As the total delay in this case and the resolutions are important metrics, the achievable rate and resolutions are discussed in the following:

When VU is not allocated a channel, it needs to generate the VR frames locally at the expense of energy consumption and resolution degeneration according to its local computing capability (CPU frequency). Let $f_{n}$ be the computation capability of VU $n$ , and it varies across VUs. Adopting the model from [20], the energy per cycle can be expressed as $e_{n,cyc}=\eta f_{n}^{2}$. Therefore, the energy consumption overhead of local computing can be derived as:

Inherently, $\mu_{n}$ is the battery weighting parameter of energy for VU $n$. The battery state of each VU can be different, then, we assume that $\mu_{n}$ is closer to $0$ with a higher battery.

In terms of the resolution, if VU $n$ is doing local generation, the resolution of the current frame will degenerate to the highest resolution that VU $n$ can process with tolerable delay. Thus, the frame resolution of VU $n$ at $t$ is formulated as:

where

Here, $J_{n}^{t}$ is the rank of the highest available resolution of VU $n$ among all resolution $\mathcal{D}$, and $J$ is the number of resolution types. $\iota$ is the duration of each time step ($\iota=\frac{1}{T}$), and $f_{n}\times\iota$ is the maximum datasize can be locally generated by VU $n$ in one step. The overall system model is shown in Fig. 1.

Different users have different purposes for use (video games, group chat, etc.). Therefore, they also have varying expectations of satisfactory FPS $\tau_{n,F}$. For each successive frame, the total delay the tolerable threshold (occurs in “Case 1”) and the insufficient resolution ($Res_{n}^{t}<G_{J}$, occurs in “Case 2”) lead to a frame failure. We set a frame success flag $I_{n}^{t}$ for VU $n$ at step $t$ as:

Our goal is to decide channel allocation and downlink power allocation for the transmission of $T$ frames. The objectives can be divided into 1) fulfill the FPS requirements of different users as possible and minimizing the total transmission failures, 2) optimize VUs’ device energy usage regarding their battery state, and 3) increase VUs’ received frames resolutions.

We define an indicator function $\chi[A]$ which takes $1$ if event $A$ occurs and takes $0$ otherwise. From the above discussion, our optimization problem is

The $\omega_{1},\omega_{2},\omega_{3}$ are the weighting parameters of frame failures, local device energy consumption, and VU devices received frames resolution. In practice, these parameters will be reflected in the reward setting III. The first part of the objective function (i.e., $\min\limits_{n\in\mathcal{N}}[(\sum_{t=1}^{T}I_{n}^{t})-\tau_{n,F}]$) is to make every VU fulfill their FPS requirements as possible, and the second part is to minimize the local energy usage and maximize the frame resolutions. Constraint $C1$ is our integer optimization variable which denotes the computing method and channel assignment for each user at every time step. Constraint $C2$ is the limit of the downlink transmission power for all VUs.

Here we describe the execution flow of the system given a solution to the optimization problem. The key idea of the two cases design (server or local generation) is to utilize the limited network resources and let some VUs spare the resources by doing local generation at each time. Thus, this paper uses a slotted time structure by applying the clock signal from the server for synchronization. Specifically, in each time slot $t$, one frame of each VU will be executed. Considering the very short duration $\iota$ of each time slot, we assume that the channel attenuation $h_{n,m}^{t}$ varies across different slots but remains the same in one slot. At the beginning of each slot $t$ (frame), the server collects the time-varying information of each VU (e.g., channel attenuation, compression rate), and sends the decisions on channel access (local or server generation, if server generation, which channel) to all VUs through a dedicated channel. Then, VUs assigned to no channel will locally generate the frame immediately, and the others will wait for the server to transmit the frames to them.

The optimization variables in the formulated problem, computation method choice, channel arrangement, and power allocation make the formulated problem highly coupled with inseparable mixed-integer non-linear programming (MINLP) optimization problems, where the discrete variable $\boldsymbol{Z}$ and continuous variable $\boldsymbol{P}$ are inseparable, which is NP-hard [21]. Moreover, this formulated problem is time-sequential, where the number of variables increases with $T$. Thus, the traditional optimization methods are unsuitable for our proposed problem due to the daunting computational complexity. Also, as the problem contains too many random variables, model-based reinforcement learning (RL) approaches that require transition probabilities are infeasible techniques to tackle our proposed problem. Heuristic search methods can possibly be a solution to sequential problems. However, it doesn’t change the approximation of the policy, while naively making improved action selections given the current value function. Besides, a huge tree of possible continuations is usually considered when using this method [22], which further makes it impractical to apply heuristic search in such a complicated scenario with a huge dimension of decisions. Therefore, it is highly necessary to design a comprehensive model-free Multi-Agent Deep Reinforcement Learning (MADRL) method to tackle the problem with heterogeneous optimization variables and distinct requirements from all VUs.

In the DRL environment, weeding out less relevant and less time-varying variables is essential. Therefore, we set two agents: $Agent_{1}$ for optimizing the channel arrangement, and $Agent_{2}$ for allocating the downlink transmission power.

The appropriate action settings that are directly related to the optimization variables are critical for finding a good solution. In this environment, the action $a_{1}^{t}$ and action $a_{2}^{t}$ are respectively the $Agent_{1}$ and $Agent_{2}$ actions, explained below.

In the traditional CTDE framework [23], the rewards are shared by different agents. However, in our scenario, the objectives and optimization variables are highly different. e.g., If VU $n$ is allocated to do the local computation, the rewards for energy consumption and resolution degeneration are irrelevant to $Agent_{2}$. Therefore, we design the rewards for $Agent_{1}$ and $Agent_{2}$ separately. Moreover, as we consider a user-centric scenario, we decompose the rewards for $Agent_{1}$ and $Agent_{2}$ among different VUs, and construct a novel algorithm in Section IV.

In contrast to decomposing the overall reward into separate sub-goal rewards as done in HRA, we built a user-centric reward decomposition Critic, which takes in the rewards of different users and calculates the actions-values separately. In other words, we give the network a view of the value for each user, instead of merely evaluating the overall value of an action based on an overall state. Simultaneously, considering the highly different roles, action spaces, and objectives of the two agents, we design the Heterogeneous Actors structure. The two agents are updated by different rewards and advantages, which will be expounded in the following.

Function process: In each episode, when the current transmission is accomplished with the channel access arrangement $a_{1}^{t}$ from $Agent_{1}$, and downlink power allocation $a_{2}^{t}$ from $Agent_{2}$, the environment will issue every VU’s rewards $\boldsymbol{R}_{1}^{t}=\{R_{1}^{t}[1],R_{1}^{t}[2],...,R_{1}^{t}[N]\}$ for $Agent_{1}$, and $\boldsymbol{R}_{2}^{t}=\{R_{2}^{t}[1],R_{2}^{t}[2],...,R_{2}^{t}[N]\}$ for $Agent_{2}$ as feedback to different VUs. The global state $s_{1}^{t}$ and the next global state $s_{1}^{t+1}$ will be sent to the user-centric Critic to generate the state values for each VU. Then, the rewards $\boldsymbol{R}_{1}^{t}$ and $\boldsymbol{R}_{2}^{t}$ with state-values will be used to calculate the advantages $\boldsymbol{A}_{1}^{t}=\{A_{1}^{t}[1],A_{1}^{t}[2],...,A_{1}^{t}[N]\}$ and $\boldsymbol{A}_{2}^{t}=\{A_{2}^{t}[1],A_{2}^{t}[2],...,A_{2}^{t}[N]\}$ for $Agent_{1}$ and $Agent_{2}$. The user-centric Critic takes in the global state and advantages for calculating the Critic losses $\{L_{1},L_{2},...,L_{N}\}$for different VU, and updates with the sum losses similar to HRA. Note that the global state ($s_{1}^{t}$), reward ($\boldsymbol{R}_{1}^{t}$), and advantage are all from $Agent_{1}$, as those of $Agent_{2}$ are mainly a portion extracted from $Agent_{1}$. In terms of the Actors, we use $\boldsymbol{A}_{1}^{t}$ and $\boldsymbol{A}_{2}^{t}$ to update Actors of $Agent_{1}$ and $Agent_{2}$, separately. The above-mentioned process is illustrated in Fig. 4.

Heterogeneous actors update function: In equation (15) (i.e., $\Delta\theta=\mathbb{E}_{(s^{t},a^{t})\sim\pi_{{\bar{\theta}}}}[\triangledown f^{t}(\theta,A^{t})]$), we established the policy gradient for PPO Actor, and our UCHA uses the sum-advantages of every VU for the update. Then, we have the gradient $\Delta\theta_{1}$, $\Delta\theta_{2}$ for $Agent_{1}$ and $Agent_{2}$ as:

where $A_{1}^{t}[n],A_{2}^{t}[n]$ denote the advantages of different VUs for $Agent_{1}$ and $Agent_{2}$. A truncated version of generalized advantage estimation (GAE) [35] is chosen as the advantage function:

where

$\bar{T}$ specifies the length of the given trajectory segment, $\gamma$ specifies the discount factor, and $\lambda$ denotes the GAE parameter. And these parameters will be specified in experiments. $V_{\phi^{\prime}}$ is the target Critic with parameters $\phi^{\prime}$, which will be periodically replaced by $\phi$. The user-centric Critic has $N$ outputs, where $V_{\phi^{\prime}}(\cdot)[n]$ means the Critic-head for VU $n$, which takes in the state, and outputs the state-value for VU $n$. Note that we both use $s_{1}^{t}$, but with their separate rewards, because $s_{1}^{t}$ is the global state, and we only give the Critic an overall view of the whole task.

User-centric Critic update function: In this user-centric Critic, we compute the value losses for each VU separately to enable UCHA to have a user-specific view. Similar to the update method in HRA [16] that uses the sum losses of different components, we use the sum losses of different VUs. Therefore, the traditional PPO-Critic loss in Eq. (17) (i.e., $L(\phi)=(V_{\phi}(s^{t})-V^{t}_{target})^{2}$) is re-formatted into:

where $V_{target}[n]=A_{1}^{t}[n]+V_{\phi^{\prime}}(s_{1}^{t})[n]$, and $L(\phi)[n]$ is the Critic loss for VU $n$.

As explained above, this user-centric Critic uses centralized training with equation (24). The computation complexity with real-time expenditure in experiments will be discussed in Section V.

As we introduce a novel UCHA algorithm with unique actor and critic structures, we weed out the key points of UCHA one by one, and design the following baseline algorithms:

• •

  Independent PPO (IPPO). The most intuitive way of using RL in such a cooperative interactive environment is to implement two independent RL agents interacting with each other. We implemented two independent PPO with user-centric Critics for optimizing the channel access and downlink power allocation. They both use a normal and separate Actor and Critic and update with their own states, actions, and rewards. This baseline is to examine the effect of the heterogeneous Actors structure in UCHA.

• •

  Heterogeneous-Actor PPO (HAPPO). We also implement an HAPPO structure that is similar to UCHA. The HAPPO does not apply the user-centric architecture, but merely uses one normal Critic that outputs a single value for the whole global state. Therefore, in HAPPO, we use the sum rewards of each Agent as the single reward. i.e., $Agent_{1}$ uses rewards $\boldsymbol{R_{1}^{t}}=\sum_{n=1}^{N}R_{1}^{t}[n]$, and $Agent_{2}$ uses $\boldsymbol{R_{2}^{t}}=\sum_{n=1}^{N}R_{2}^{t}[n]$. This baseline is to testify the effect of the User-centric structure of the Critic.

• •

  Random. The two random Agents select actions randomly, which represents the system performance if no optimization strategy is performed. The random policy serves as a naive baseline to show the results if no optimization has been conducted.

We introduce a set of metrics to evaluate the effectiveness of our proposed methods.

• •

  Worst VU frames. We define the Worst VU frames, i.e., $\min\limits_{n\in\mathcal{N}}(\sum_{t=1}^{T}I_{n}^{t}-\tau_{n,F})$ in Eq.(9). The “Worst VU” means the VU has the biggest gap between its successfully received frames and its required FPS, and the “Worst VU frames” refers to this gap.

• •

  Successful FPS. The number of successful frames among total $T$ frames determines the FPS of the VR frames and hence fluidity of the Metaverse VR experience.

• •

  The sum local energy consumption. We first illustrate the sum energy consumption during training, and then investigate it for different VUs in detail.

• •

  The received frame resolutions. The received frame resolutions of different VUs will be shown to testify the comprehensive view of our proposed methods.

When $z_{n}^{t}=m$, the channel attenuation becomes a very important variable for the downlink rate. In this paper, the channel attenuation is simulated as $h_{n,m}^{t}=\sqrt{\beta_{n}^{t}}g_{n,m}^{t}$. The Rician fading is used as the small-scale fading, $g_{n,m}^{t}=\sqrt{\frac{K}{K+1}}\bar{g}_{n,m}^{t}+\sqrt{\frac{1}{K+1}}\tilde{g}_{n,m}^{t}$, where $\bar{g}_{n,m}^{t}$ stands for the Line-Of-Sight (LOS) component, and $\tilde{g}_{n,m}^{t}$ means the Non-LOS (NLOS) component that follows the standard complex normal distribution $\mathcal{C}\mathcal{N}(0,1)$. The large-scale fading $\beta_{n}^{t}=\beta_{0}(L_{n})^{-\alpha}$, and $L_{n}$ represents the distance between the $n$th VU and server. $\beta_{0}$ denotes the channel attenuation at the reference distance $L_{0}=1$ m. The path-loss exponent $\alpha$ is simulated as $2$, and the Rician factor $K$ is simulated as $3$. Note that as the duration of each time slot is too short, we don’t consider the geographical movements of VUs in each time step.

Consider a $30\times 30$ $m^{2}$ indoor space where multiple VUs are distributed uniformly across the space. We set the number of channels to be $3$ in each experiment configuration, and the number of VUs to be from $5$ to $8$ across the different experiment configurations. To simplify the simulation, we use the most common standard of the resolutions but ignore the various horizontal FoVs and aspect ratios for different VR devices. We set the resolutions of one frame can be 1440p ($2560\times 1440$, which is known as Quad HD), 1080p ($1920\times 1080$, which is known as Full HD), 720p ($1280\times 720$, which is known as HD), and below (considered as insufficient resolution). Each pixel is stored in 16 bits [36] and the factor of compression is selected in the uniformly random distribution $[300,600]$ [18]. And each frame consists of two images for two eyes. Therefore, the data size of each frame can be $D_{n}^{t}\in\{\frac{2560\times 1440\times 16\times 2}{compression},\frac{1920\times 1080\times 16\times 2}{compression},\frac{1280\times 720\times 16\times 2}{compression}\}$. The maximum flashed rate, $T$ frames in one second is taken to be 90, which is considered a comfortable rate for VR [4] applications. The bandwidth of each channel is set to $10\times 180$ kHZ. The required successful frame transmission count $\tau_{n,F}$ is uniformly selected from $[60,90]$, which is higher than the acceptable of $60$ [4]. The maximum powers of the server are $100$ Watt. The server computation frequency is $10$ GHz and the computation capabilities of each VU are selected uniformly from $[0.3,0.9]$ GHz. For all experiments, we use a single NVIDIA GTX 2080 Ti and $2\times 10^{5}$ training steps, and the evaluation interval is set to be $500$ training steps. As there are several random variables in our environment (e.g., channel attenuation, compression rates), all experiments are conducted under ten different global random seeds, and the error bands are drawn to better illustrate the model performances.

Reasonable and comprehensive experiments are of vital importance in such a complicated scenario. In this section, we will illustrate and analyze some evaluated metrics completely during training. For brevity, we show the performances of different models in two experimental configurations: the less complicated scenario with 6 VUs and the more complicated one with 8 VUs. And the overall results of every scenario are shown in Table I. We also analyze the computational complexity, and then the value losses of each VU are illustrated to give a more “user-specific” analysis. Note that to better evaluate the different metrics, we do not apply early termination if the task fails in the evaluation stage.