Delay-Effective Task Offloading Technology in Internet of Vehicles: From the Perspective of the Vehicle Platooning

The mobile edge computing (MEC) functions can be realized by integrating edge servers of RSUs/MBSs. In this way, the data processing and storage can be performed directly on these servers and the processing delays can be reduced. Subsequently, in [6], Luo et al. deployed RSUs based edge computing along roads to provide enough bandwidth for computing task offloading requiring by vehicles, for the purpose of minimizing the data processing costs while ensuring delay constraints. In particular, they modeled the data scheduling as a DRL problem, which can be solved by an improved deep Q-network (DQN) algorithm. By integrating the resources of MBS and RSU, In [7], Sonmez et al. proposed a vehicle edge orchestrator based on two-stage machine learning, under the constraints of the task offloading success probability and task processing delay. Similar works were also done in [8].

Due to the high-speed mobility of the vehicles, the effective communication duration between the vehicles and MBSs/RSUs cannot be enough to support achieve task offloading, since the corresponding processing delay may be higher than the effective communication duration.

Achieving task offloading by exploiting the computational resources of multiple vehicles is effective. For example, by offloading the task to the vehicles moving along the same direction of the road, it is possible to achieve task offloading while ensuring a longer effective communication duration. In [9], Qiao et al. regarded the moving vehicles as mobile edge computing resources with the purpose of establishing a collaborative distributed computing architecture, under which a collaborative task offloading and output transmission mechanism was proposed to ensure the processing delay and application-level performance. In [10], Xiong et al. proposed a heterogeneous intelligent task offloading framework based on three types of V2X communication technologies.

Considering the computational resources provided by the vehicle edge computing (VEC) server, local vehicles and other vehicles, to enhance the delay benefits and energy cost benefits, in [11], Huang et al. proposed a task offloading and resource allocation method by jointly designing of task type and vehicle speed, under which the utility function of the vehicle can be derived based on the energy costs and task execution revenue. Finally, a joint optimization scheme for task offloading and resource allocation was developed to maximize the utility level of vehicles constrained by the processing delay, computing resources, and wireless resources. Similar works were also done in [12].

Above works have considered the computational resources supported by the vehicles, and provided an available solution to deal with task offloading interrupt existing in offloading centered on the RSUs. But the coordinated task offloading framework among vehicles and collaboration were ignored.

Different from randomly moving vehicles, the vehicles in VEH-PLA are featured by the same driving behaviors, such as same driving direction, smooth speeds, and tight following distance [13]. As an innovative method through V2V communication technology that enables the coordinated operation among multiple vehicles, the VEH-PLA offers a stable topology to provide a larger size of computational resources and effective communication duration [14]. In addition, the task offloading by using the vehicular resources can further decrease the corresponding processing delay.

In [13], Xiao et al. redistributed complex tasks via VEH-PLA, thereby optimizing the utilization of VEH-PLA’s idle resources for enhanced task processing. This approach benefited from the inherent stable connectivity, adjustable maneuverability, and sufficient charging capabilities characteristic of VEH-PLA. Moreover, they introduce a Stackelberg game model to describe the interplay between mobile edge platoon cloud (MEPC) platforms and vehicles requiring task offloading, focusing on maximizing their combined utility. In [15], Zheng et al. analyzed the processing delay and energy consumption of local offloading and VEH-PLA offloading when offloading the task to VEH-PLA members, and proposed a method to minimize the long-term maximum task queue by optimizing the allocation of communication and computing resources. Finally, the effectiveness of the proposed method was demonstrated through online algorithms using Lyapunov optimization and successive convex approximation (SCA) methods. In [16], Chao et al. designed a contract based incentive mechanism to incentivize vehicles to form VEH-PLA and share their computing resources. Then, for maximizing vehicle utility, a joint optimization of offloading decisions, communication, and computing resource allocation schemes of the vehicle and BS was formulated. The problem of task offloading failure caused by potential disconnections among vehicles was studied in [17], a multi-leader multi-follower Stackelberg game was introduced for the co-optimization of pricing strategies and computational resource distribution, with the goal of maximizing participant revenues. Similar works were also done in [18].

However, for one thing, all above studies ignored wireless interference over the processing of task offloading, which affects the QoS of communication channel and the processing delay, resulting a failure of the task offloading. For another thing, facing a large number of task offloading requests, there has been no analysis of task types, particularly the differences between tasks managed in a queue environment and those performed by traditional road vehicles. VEH-PLA, as a new type of computational resource allocation framework, allows its members to drive collaboratively, collectively completing public tasks such as environmental perception, decision-making, and emergency response. This can effectively reduce the frequency of task offloading and alleviate the pressure of insufficient computing resources in the IoV.

Without loss of generality, we employ a Cartesian coordinate system and define the positive direction as moving towards the right of the Fig. 1, with coordinates measured in meters. The position of $i$-th vehicle at time slot $t$ is $\textbf{z}^{i}_{ve}(t)=[x_{ve_{i}},y_{ve_{i}}]^{\textbf{T}}$. Similarly, the locations of $j$-th RSU and $k$-MBS can be denoted by $\textbf{z}^{j}_{rsu}(t)=[x_{rsu_{j}},y_{rsu_{i}}]^{\textbf{T}}$ and $\textbf{z}^{k}_{mbs}(t)=[x_{mbs_{k}},y_{mbs_{k}}]^{\textbf{T}}$, respectively.

Using a truncated Gaussian distribution model to simulate vehicle traffic is more in line with the actual movement characteristics of vehicles [19], where the mean is $\mu$ and the standard deviation is $\sigma$. Let $v_{\text{min}}=\mu-3\sigma$ and $v_{\text{max}}=\mu+3\sigma$ be the minimum and maximum truncation ranges for speed, and let the speed be represented by $v_{i}$. To ensure the consistency of speed among members of VEH-PLA, the generated speeds need to be synchronized. In this way, the speed of member-vehicles, denoted by $v_{\bar{S}_{ve}}$, is set to the average speed of the whole platooning, namely $v_{\bar{S}_{ve}}=\frac{1}{m_{\rm ve}}\sum^{m_{\rm ve}}_{i=1}v_{i}$.

Vehicles driving on the same road usually have similar task processing requirements (e.g., the road condition information, road congestion information, and weather information etc). The framework VEH-PLA has a stable topology structure and tight following distance. Accordingly, on the one hand, tasks requesting from vehicles in the platooning can be regarded as public tasks, and can be handled uniformly by using the same resources. While the individual differentiated task requests can be called as personalized tasks, which need allocate additional resources to process the tasks. On the other hand, the VEH-PLA framework also extends the available computing resources of individual vehicle, increasing the success probability of task offloading. Based on the advantages of VEH-PLA, when the vehicle requests for a task offloading, based on the constraints of the task types, size of available computing resources and processing delay provided by different offloading models, the vehicle will select the best offloading decision among the LOCAL offloading, VEH-PLA offloading, RSU offloading, and RELAY offloading, as shown in Fig. 3.

In detail, the VEH-PLA offloading model can support two kinds of task types, i.e., public tasks and personalized tasks. On the one hand, the vehicle requesting for a task offloading belongs to the platooning, it first determines whether the task is public or not. If the task is public, it does not occupy any resources to compute the task and directly gets the result from the lead vehicle of the VEH-PLA. Otherwise, it will perform task offloading. On the other hand, when the $i$-th vehicle does not belong to the platooning, if effective communication duration between $ve_{i}$ and any one of vehicles in the platooning is less than or equal to the affordable offloading delay of $i$-th vehicle, and the latter has enough computing resources satisfying the demand of $i$-th vehicle’s task, then $ve_{i}$ can execute the VEH-PLA offloading model.

As a result of these diverse offloading models, differentiated demand of vehicles, different scale of computing resources, the processing delay of a task offloading consists of different types of delays for LOCAL, VEH-PLA, RSU and RELAY offloading models.

Based on the above description, the purpose of this paper is to optimize the task offloading decisions of all vehicles demanding for a task offloading at episode $t$, denoted by

Formally, the optimization problem of a delay-effective task offloading can be written as follows

where constraints of Eq. (8b) and Eq. (8c) indicate that the $k$-th vehicle with demand of a task offloading only chooses one of four available offloading models at episode $t$, as described in Subsection III-C. Eq (8d) ensures that the processing delay satisfies the maximum affordable delay requirements of $k$-th vehicle, while Eq. (8e) can achieve the load balancing of the resources on the RSUs, by allowing to serve $N$ vehicles simultaneously at most over all episode.

To obtain a satisfactory task offloading decision, a search must be required in the decision space with the size of ${{(n_{\text{ve}}+n_{\text{rsu}}+n_{\text{mbs}})}^{n_{\text{ve}}}}$. Based on this fact, traditional methods, such as [23], typically necessitate adjusting decisions iteratively to achieve the global optimization, but do not work well in rapidly moving vehicular environments. Facing the challenging in acquiring a priori knowledge of the dynamic environment, the DRL has been shown that it allows for learning optimal offloading schemes via continual interaction with the stochastic vehicular environments [24]. Therefore, in Section IV, based on a CTDE, we present a multi-agent DRL (MADRL) algorithm to strike a balance between computational complexity and convergence performance to find a long-term optimal solution, satisfying the processing delay and resource load balancing requirements.

First of all, each agent in IoV can collect or observe the local information in its surrounding environment at episode $t$. Subsequently, the agent independently makes an action choice for maximizing the overall expected shared reward. Formally, based on Dec-POMDP, a distributed vehicular task offloading can be represented as a tuple of $\langle I,S,\{O_{i}(t)\}_{i\in I},P,\{A_{i}(t)\}_{i\in I},\{r_{i}(t)\}_{i\in I},\gamma\rangle$ where $I$ is the set of agents, $S$ represents the set of all possible states of the environment, including all states that the system can reach at any episode, $s(t)\in S$ represents the environmental state at episode $t$, $O_{i}(t)$ is the local observation space possessed by $i$-th agent at episode $t$, $o_{i}(t)\in O_{i}(t)$ is the specific instance of the local observation, $P$ is the state transition function, $A_{i}(t)$ is the action spaces available for $i$-th agent at episode $t$, $a_{i}(t)\in A_{i}(t)$ represents the action selected by $i$-th agent, $r_{i}(t)$ is the reward of $i$-th agent at episode $t$, and $\gamma\in[0,1]$ is the discount factor that balances the present and future rewards. Due to limited observations achieved by each agent, it cannot fully observe the environmental state space $S$ and instead use the local observation $o_{i}(t)\in\{O_{i}(t)\}_{i\in I}$ as a substitute for the state $s(t)\in S$. In addition, the state transition function can be ignored, namely $P$ can be removed from the tuple, since the task offloading decision problem is modeled as a distributed MADRL task. Therefore, the configurations of distributed vehicular task offloading are rewritten as follows.

1) Local Observations: In terms of the location, task size, computing resource demand, processing delay, and relationship with vehicular platooning, the local observation of $i$-th agent at episode $t$ can be represented as

where $\Theta_{i}(t)$ denotes the location of $i$-th agent at episode $t$, $\lambda_{i}(t)$ represents the task size, $\lambda_{i}(t)\rho_{i}(t)$ signifies the computing resources needed by the task, $D_{{ve}_{i}}^{{ve}_{i}}(t)$ is the corresponding processing delay required for executing LOCAL offloading, and $\delta_{{ve}_{i}}^{{\rm pla}_{ve_{j}}}(t)\in\{0,1\}$ indicates that whether $i$-th agent makes a connection with $j$-th agent belonging to the vehicular platooning. If the $i$-th agent cannot be connected to any agents existing in the VEH-PLA, then $\delta_{{ve}_{i}}^{{\textrm{pla}}_{ve_{j}}}(t)=0$.

2) Action Space: The action space of the $i$-th agent consists of potential offloading decisions. In this way, the set of offloading decisions available to the $i$-th agent at episode $t$ can be represented as

3) Reward: To update task offloading decisions to maximize the whole system rewards, all agents share a common team reward. By using similar methods to deal with actions of agents that cannot satisfy the constraints [25], we set a negative penalty for agents to describe the results when violating constraints of Eq. (8b) - Eq. (8e). Then the reward for the $i$-th agent based on the objective function and constraints to shape the training process is given by

where $pc$ is the penalty coefficient, while $\eta_{1}$ and $\eta_{2}$ are penalties corresponding to constraints of Eq. (8d) and Eq. (8e), respectively. A well-designed penalty structure for the training process is crucial to accelerate the algorithm convergence.

Under the framework of the cooperative MADRL, the agents collectively execute task offloading decisions, with the purpose of the whole system reward maximization, i.e., optimizing the global reward. Mathematically, the global reward, denoted by $R^{\text{tot}}$, can be represented as

However, the global reward $R^{\text{tot}}$ is obtained only at the end of the episode, resulting in a low training efficiency. To solve this problem, we extract the rewards of all agents in each decision episode to as the instantaneous global reward

Furthermore, in the context of Dec-POMDP, the scheduling of computing tasks in each decision episode aims to maximize the whole system reward. In addition, the decisions provided by the agents are also effected by each other. Accordingly, above computing task scheduling process can be regarded as an exact potential game (EPG) [26], which can be defined as a scenario that the change of individual agent’s utility by updating its offloading strategy, and be exactly equivalent to the change of all system’s potential utilities. With the help of EPG, we conclude that the instantaneous reward maximization of an individual agent can lead to the maximization of the whole system’s reward. That is, when the instantaneous reward of $i$-th agent, i.e., $r_{i}(t)$, is maximized, the total rewards of all agents in each decision episode, i.e. $r^{\text{tot}}(t)$, is also maximized. Mathematically, the instantaneous global reward can be maximized by updating offloading decisions of all agents as follows

where $\textbf{a}(t)=\{a_{1}(t),a_{2}(t),\ldots,a_{I}(t)\}$ and and $\textbf{o}(t)=\{o_{1}(t),o_{2}(t),\ldots,o_{I}(t)\}$ denotes the set of actions determined by all agents and the set of local spaces observed by all agents at episode $t$, respectively.

Given a local observation of $i$-th agent at episode $t$, i.e., $o_{i}(t)$, the expected reward when it selects the action $a_{i}(t)$ can be calculated by the corresponding Q-value function. That is, with the configurations of $o_{i}(t)$ and $a_{i}(t)$, the expected reward of $i$-th agent can be represented as

where $\textbf{E}[\cdot]$ is the expectation function, and $\pi$ is a policy rule that guides agents for choosing corresponding actions based on their local observations.

Due to the dynamic nature and uncertainty of the environment, it is difficult to derive the optimal Q-value function for each agent. To solve this problem, we introduce the Bellman equation into the decision-making process of the agents, since it can simplify multi-stage decision problems into a series of single-state problems by using recursive decomposition. In this way, the optimal Q-value function, denoted by $Q^{\ast}(o,a)$, can be derived, which denotes the maximum expected reward by taking action $a$ within a given local observation $o$ and following the optimal policy, the expression is shown in Eq. (16).

Furthermore, by implementing a greedy selection mechanism based on the optimal Q-value function of $Q^{\ast}(o_{i}(t),a_{i}(t)$ in Eq. (16), which prioritizes the action that is anticipated to provide the highest reward in the current state, thereby we can derive the corresponding optimal policy for the local observations, which is given by

However, finding the optimal policy by using Eq. (17) ignores the potential long-term benefits of actions that do not be the optimal in the current context, resulting in an inadequate exploration for potential long-term benefits. To solve this problem, we introduce a $\epsilon$-greedy strategy, under which the agent can either select the best action known currently with the probability of $1-\epsilon$, or choose any feasible action randomly with the probability of $\epsilon$.

Under the condition of limited strategy space for all agents, the EPG can converge to a pure strategy Nash equilibrium, so the expected maximum cumulative reward can be obtained. However, finding the global optimal action of a task offloading strategy is a challenging problem, due to the dynamic environment in which the agents. In the next subsection, we utilize a MADRL method based on CTDE to search for an approximate optimal action for the EPG.

The CTDE, as a cutting-edge framework that integrates a hybrid MARL framework, allows for centralized training of agents using global state information, while enabling agents to act in a decentralized manner based on local observations. However, extracting the policies from the centralized training process that can be decentralized for execution remains a challenge for implementing CTDE in MADRL, since the discrepancy between global information used during training and local observations available during execution leads to issues of policy effectiveness and stability. To overcome this challenge, the VDN algorithm is introduced to solve the problem of aligning decentralized execution with centralized training outcomes, from the perspective of facilitating individual agent decisions that collectively contribute to achieving the same global optimal policy.

Therefore, in this section, based on the framework of CTDE-MADRL, we propose an effective task offloading method empowered by VEH-PLA, called as a heterogeneous computational resources allocation (CTDE-MADRL-HCRA), which integrates the advantages of VEH-PLA coordination, dynamic learning capability of MADRL, efficient collaboration mechanism of CTDE strategies, and decision optimization techniques of VDN to address challenges of online offloading.

To narrow the gap between global state information and local state information, the method of storing historical observations as an auxiliary information to construct the Q-value function model has been proven to be effective [26]. Accordingly, we adopt $\bm{\tau}(t)=\{\tau_{1}(t),\tau_{2}(t),...,\tau_{I}(t)\}$ to represent the collection of historical observations of all agents, where $\tau_{i}(t)=\{(a_{i}(1),o_{i}(1),r_{i}(1)),...,(a_{i}(t-1),o_{i}(t-1),r_{i}(t-1))\}$ is joint action-observation history of $i$-th agent.

The CTDE-based framework aims to learn a joint Q-value function $Q_{\text{tot}}(\bm{\tau}(t),\bm{a}(t))$ with global state information by using centralized training process. Once $Q_{\text{tot}}(\bm{\tau}(t),\bm{a}(t))$ is learned, which cannot be obtained by using distributed DRL algorithms, distributed agents based on CTDE can make more reasonable independent decisions based on individual local observations. The proposed CTDE-based computational offloading architecture is shown in Fig. 4, and can address the aforementioned challenges. It can be noticed that the architecture consists of individual agent modules, a central training mechanism, and a policy execution framework. In detail, individual agent modules are responsible for processing local observations and selecting corresponding actions based on their current policies. The central training mechanism can solve the challenge of learning a joint Q-value function that incorporates global state information, enabling the optimization of collective outcomes. The policy execution framework makes it possible for agents to act independently in a decentralized manner during execution, relying on their local observations while still aligning their actions with the learned global objectives.

To achieve the decentralized executions executed from centralized training outcomes, based on the VDN technology, we can decompose the joint Q-value function $Q_{\text{tot}}(\bm{\tau}(t),\bm{a}(t))$ at episode $t$ into the Q-value functions of individual agent, denoted by $Q_{i}(\tau_{i}(t),a_{i}(t)),i\in I$. That is,

In practical application, constructing or maintaining directly a Q-value table for each state-action pair is impractical, as the state space gets very large with the increasing of agents, making the Q-value function difficult to be stored or computed directly. Therefore, we use a deep Q-value network (DQN) to approximate the above function, that is, we adopt $Q_{i}(\tau_{i}(t),a(t))\approx Q_{i}(\tau_{i}(t),a_{i}(t);\theta_{i}(t))$, where $\theta_{i}(t)$ is the parameter of $i$-th agent’s Q-value network. Therefore, Eq. (18) can be rewritten as

In this way, the decomposition from joint Q-value function to that of individual agent can be simplified for the learning task. Furthermore, with the help of VDN technique, the joint Q-value function $Q_{\text{tot}}(\bm{\tau}(t),\bm{a}(t))$ obtained from centralized learning can be embedded in the distributed execution process. In particular, to maintain the consistency in decision-making process, it is necessary to ensure that the results obtained by globally optimizing the joint Q-value function $Q_{\text{tot}}(\bm{\tau}(t),\bm{a}(t))$ (i.e., the global argmax operation) are consistent with the results set obtained by independently optimizing the Q-value function $Q_{i}(\tau_{i}(t),a_{i}(t))$ for each $i$-th agent, i.e.,

To facilitate the agents’ learning from their interaction histories and enhance decision-making accuracy, the historical observations $\tau_{i}(t)$ can be stored in a Gated Recurrent Unit (GRU) and serve as a valuable supplementary information for enhancing the agent’s Q-value function model. By minimizing the difference between the joint target Q-values and their current joint Q-values, we can obtain the optimal joint Q-value function. Let $Q_{i}^{-}$ be the target Q-value of $i$-th agent. Based on the result of Eq. (19), we get $Q_{\text{tot}}^{-}(\bm{\tau}(t),\bm{a}(t))\approx\sum_{i\in I}Q_{i}^{-}(\tau_{i}(t),a_{i}(t);\theta_{i}(t))$. Accordingly, the loss function of $Q_{\text{tot}}$ is defined as

where $\sigma$ is the difference between a difference target $y_{\text{tot}}$ and a joint Q-value $Q_{\text{tot}}$. The temporal difference target is an estimate by an agent of future rewards obtainable from current state and action, denoted as $y_{\text{tot}}^{x}(t)=r^{x}(t)+\gamma\max_{a(t+1)}Q_{\text{tot}}^{-}(\bm{\tau}(t+1),\bm{a}(t+1))$, $X$ is number of samples.

The loss function of the joint Q-value function in Eq. (21) cannot properly deal with the gradient estimation, since the interaction among multiple agents leads to a non-stationary target for optimization. With the help of the back propagation method, we can obtain an implicit update of individual agent networks based on global rewards rather than single reward of each agent. Furthermore, by updating individual Q-value functions of the agents and sharing the weights of the agent networks, the training complexity can be further simplified. To sum up, the pseudo-code of CTDE-MADRL-HCRA is shown in Algorithm 1, where $T_{\max}$ denotes the maximum number of epochs in the whole training, $t_{\max}$ is the maximum number of episodes in each epoch. Within each episode, all agents engage in a complete interaction with the environment. Following each epoch, sample collection and gradient updates are performed.

In this section, we validate the effectiveness and correctness of the proposed method through Python 3.11 and TensorFlow 2.14.0 simulator. In detail, a bi-directional lane with the length of 1200 meters is constructed, and the vehicles’ position are reset to the starting point when they reach the end of the road for avoiding offloading disruptions caused by the vehicles leaving the environment. Three RSUs are fixed at the position of 200m, 600m, and 1000m, and 15 moving vehicles are randomly distributed. To reduce the size of the action space and achieve load balancing, if the number of vehicles served by the same RSU is above $N$, the RSU will no longer provide the sources to other vehicles with the demand of task offloading, which can be transferred to the adjacent RSUs. In addition, the related simulation parameters are given in Table I. In addition, to model the task categorization (i.e., public tasks and personalized tasks) for the VEH-PLA members and non-platooning members, the ratio of public tasks for the latter is 20%, while that of VEH-PLA members is set to 30%, since the vehicles in VEH-PLA usually have similar task requirements.

In the training processing of the DRL model, the discount factor $\gamma$ is set to 0.9 for reflecting the relative importance of future rewards to immediate ones. The learning rate $\alpha$ is determined to be 0.0003 to regulate the pace of model weight updates, ensuring stability and convergence speed in the learning progression. The exploration factor $\epsilon$ is initially set to 1 and decays exponentially with an increase in exploration attempts, diminishing ultimately to 0.05. These settings allow the model to extensively explore potential action strategies initially and gradually focus on enhancing the best strategies discovered. A buffer with the size of 2000 provides the learning model with an ample experiential reservoir. An update interval for the target, occurring once every 100 episodes, contributes to the stability of target values during learning. Furthermore, the mini-batch size is set to 64, optimizing the efficiency of the training process and data utilization.

To validate the effectiveness and correctness of task offloading decisions based on VEH-PLA proposed in the paper, i.e., CTDE-MADRL-HCRA, which is based on VDN, VEH-PLA and task categorization, we adopt offloading baseline schemes based on VDN, IQL and QMIX, respectively. In particular, with the consideration of the VEH-PLA and task type or not, baseline schemes can be classified as follows.

• •

  [27]: Offloading schemes based on the VDN, and these based on VEH-PLA and task categorization, denoted by VDN-NoVP-NoTC, VDN-VEH-PLA and VDN-TC;

• •

  [28]: Offloading schemes based on the QMIX, and along with both VEH-PLA and task categorization, denoted by QMIX-NoVP-NoTC, QMIX-VEH-PLA-TC;

• •

  [29]: Offloading schemes based on the IQL, and along with both VEH-PLA and task categorization, denoted by IQL-NoVP-NoTC and IQL-VEH-PLA-TC;

• •

  [30]: LOCAL offloading;

• •

  [31]: LOCAL and RSU offloading.

In the following, we investigate the impact of the numbers of episodes and vehicles, the sizes of VEH-PLA and task data on the convergence rate, average reward, and processing delay.

With the settings of 15 vehicles, consisting of two VEH-PLA with 5 vehicles respectively, and 5 non-platooning vehicles, other simulated parameters are set based on Table I, Fig. 5 demonstrates the relationship between the accumulated average reward nd the number of training episodes. On the one hand, it can be noticed that, with the increment of the number of training episodes, CTDE-MADRL-HCPA, along with other baseline schemes, shows a fast convergence speed and good stability, which validates the effectiveness analyzed in Subsection IV-C. On the other hand, the average rewards of CTDE-MADRL-HCRA, as well as QMIX-VEH-PLA-TC and IQL-VEH-PLA-TC employing the offloading model proposed in this paper, are significantly higher than these of baseline schemes. The corresponding reason is that the VEH-PLA and task categorization reduce the number of task offloading requests, and further reduce computational resource consumption of vehicles belonging to VEH-PLA, which can be used to satisfy the demand of task offloading requiring by other vehicles. In particular, as shown in Table  II, compared with VDN-NoVP-NoTC [27], QMIX-VEH-PLA-TC [28], QMIX-NoVP-NoTC [28], IQL-VEH-PLA-TC [29], and IQL-NoVP-NoTC [29], average reward achieved by CTDE-MADRL-HCPA is increased by 11.45%, 2.38%, 12.86%, 4.96% and 14.25%, respectively, which suggests that the CTDE-MADRL-HCRA possesses a superior offloading decision optimization capabilities by using task categorization and VEH-PLA, making it particularly suitable for IoV systems that demand high stability and efficiency.

Next, with the same parameter settings of Fig. 5, average processing delay of task offloading is considered, since the processing delay in Eq. (8a) cannot fully reflect the exploratory nature of individual vehicle decisions within the processing of DRL, while the average processing delay can smooth out this episodic variability, offering a more consistent measure for assessing the overall system efficiency. Consequently, in Fig. 6, we observe the influence of the number of episodes on the average processing delay. It can be observed that, apart from LOCAL offloading scheme [30], the average processing delay is relatively low in the initial phase of other schemes. This is because that, in the initial phase, the computational resources of RSUs and vehicles in VEH-PLA are not occupied by vehicles requesting for task offloading, and thus have abundant computational resources. Accordingly, the tasks can be offloaded directly successfully. While in LOCAL offloading scheme, the computational resource of individual vehicle is less than these of other offloading schemes, resulting in a larger average processing delay. However, as the number of episodes increases, more task offloading requests exist and available computational resources become less, resulting in a gradual rise in average processing delay. Among all schemes, CTDE-MADRL-HCRA has the lowest average processing delay followed by the QMIX-VEH-PLA-TC scheme [28], which further demonstrates the effectiveness of proposed offloading model based on EH-PLA. On the average, compared with other offloading schemes VDN-NoVP-NoTC [27], QMIX-VEH-PLA-TC [28], IQL-VEH-PLA-TC [29], the processing delay achieved by CTDE-MADRL-HCRA is decreased by 14.52%, 6.46% and 13.52% at least, respectively. In addition, compared to the offloading schemes based on IQL (i.e., IQL-VEH-PLA-TC, IQL-NoVP-NoTC [29]), which independent updating policies of multiple agents lead to difficulties in coordinating the optimization of average processing delay, both CTDE-MADRL-HCRA and QMIX-VEH-PLA-TC are more suitable for cooperative environments and therefore exhibit a more better performance.

Furthermore, for the better observation, the results are averaged with the number of episodes varying from 500 to 1000. Considering different number of non-platooning vehicles and other simulated parameter settings keep the same as before, Fig. 7 illustrates the performance comparison of all offloading schemes in terms of the average processing delay. It can be noticed that from Fig. 7 the average processing delay increases with the increment of the number of vehicles. On the one hand, CTDE-MADRL-HCRA shows the best performance with the consideration of VEH-PLA and tesk categorization. On the other hand, as shown in Table  III, compared with CTDE-MADRL-HCRA and QMIX-VEH-PLA-TC, the growth rate of IQL-VEH-PLA-TC is about 5%. Compared with offloading schemes based on IQL, such as IQL-VEH-PLA-TC [29], average processing delays achieved by CTDE-MADRL-HCRA and these based on QMIX-VEH-PLA-TC [28] increase more stably, which can be attributed to the fact that, for offloading schemes based on the IQL, the vehicles only learn independently without cooperation. When the number of vehicles becomes large, offloading schemes based on VDN and QMIX perform more effectively.

Considering the case of two VEH-PLAs with the number of moving vehicles varying in the range of [3, 4, 5, 6, 7], and the case of no-VEH-PLA (i.e., the number of vehicles in VEH-PLA is 1), Fig. 8 illustrates the relationship between the size of VEH-PLA and average processing delay, where the number of vehicles represented at x-axis is that of a single VEH-PLA. It can be noticed that with the increment of VEH-PLA members, on the one hand, the average processing delay shows a downward trend. This is because the fact that over the size of VEH-PLA increasing, the vehicles with the demand of public task offloading can directly utilize the results of the VEH-PLA, which can avoid overlapping offloading for the same public task, decreasing the average processing delay. On the other hand, total decrement of average processing delay between any two number of VEH-PLA members becomes less significant. In the CTDE-MADRL-HCRA scheme, the average processing delays with the size of VEH-PLA being 5, 6, 7 are 2.254s, 2.223s, and 2.211s, respectively. Compared with average processing delays achieved by 5 and 6 VEH-PLA members, that of VEH-PLA with 7 members is decreased 1.44% and 1.91%. One possible is that the marginal benefit of adding additional vehicles to the VEH-PLA diminishes as the number of vehicles increases, indicating a plateau in efficiency gains. Therefore, in the case of Fig. 8, maintaining a VEH-PLA with the size of 5 or 6 members is a more appropriate choice, in terms of average processing delay, VEH-PLA stability and offloading feasibility.

The number of moving vehicles on the road is set to 15, and other parameters are set as before. I particular, considering 12 VEH-PLA members, which are classified into 1, 2, 3, 4, and 6 platooning respectively, and 3 non-platooning members, Fig. 9 describes the performance comparison of average processing delay under different number of platooning, The results clearly indicate that CTDE-MADRL-HCRA is more applicable for various platoon modes, followed by QMIX-VEH-PLA-TC, the corresponding reason is similar as that of Fig. 5, Fig. 6, and Fig. 7. Similarly, IQL-VEH-PLA-TC exhibits the worst performance. Given a certain number of moving vehicles, decreasing the number of VEH-PLA (i.e., increasing the number of vehicles in a single VEH-PLA) can have actually helped reduce the average processing delay. This is attributed to the fact that more fewer VEH-PLA can lead to more fewer number of public task offloading. However, the size of VEH-PLA is not more larger the better, one possible reason is that increased communication overhead and coordination difficulties among a larger number of vehicles can lead to inefficiencies and delays in the platooning network, and has been concluded in [32], [33]. Therefore, in this case, keeping the size of VEH-PLA with the number of 6 members can make a better trade-off between the average processing delay and platooning stability.

The size of of task data varies in range of [1.0, 2.0]Mbits, and other parameters are set as before. Fig. 10 validates the impact of task size on the average processing delay. CTDE-MADRL-HCRA shows the best performance as expected. In particular, given a task data with the size of 2 Mbits, compared with QMIX-VEH-PLATC and IQL-VEH-PLA-TC, the average processing delay achieved by CTDE-MADRL-HCRA is decreased 110 ms and 240 ms, respectively.