Collaborative Optimization of Wireless Communication and Computing Resource Allocation based on Multi-Agent Federated Weighting Deep Reinforcement Learning

To accurately simulate real and complex wireless communication scenarios, we consider the Log Distance Path Loss (LDPL) model [10] for our system. This model takes into account the random shadowing effects caused by obstacles such as hills, trees, and buildings, in different environments, to capture a broader range of propagation losses. Therefore, the path loss model between the BS and the $k$-th UE in the $i$-th time slot $t_{i}$ can be formulated as follows:

where $PL\left(d_{0}\right)$ called close-in reference distance, is obtained by using Friis path loss equation. The variable $\lambda$ exists in the path loss equation to account for the effective aperture of the receiving antenna, which is an indicator of the antenna’s ability to collect power. $d_{0}$ represents field measurements distance, that specifically denotes a value of 1m. $\gamma$ is the path loss exponent (PLE) that depends on the type of environment. $\chi$ is a zero-mean Gaussian distributed random variable with standard deviation $\sigma$ expressed in dB, used only when there is a shadowing effect. Then, the signal-to-noise ratio (SNR) received by the $k$-th UE can be expressed as follows:

where $P_{k}$ represents transmission power, the noise power spectral density is denoted as $\mathcal{N}_{0}$, and the $k$-th UE bandwidth is denoted as $b_{k}$. Consequently, the achievable rate of the distributed system during the $i$-th time slot can be expressed as:

The system takes into account the broadband transmission demands of the future wireless network and investigates the enhancements of MAFWDRL in terms of multi-parameter adjustment performance. Consequently, we consider SU-MIMO in the wireless communication system. Herein, the BS is equipped with $\lambda_{0}$ antennas, while the $k$-th UE possesses $\lambda_{k}$ antennas. During the $i$-th observation slot $\tau$, the channel transmission model of the $k$-th UE transmitting data can be represented as follows:

where $\mathrm{Y}_{\lambda_{k}}$ represents the received data of the $\lambda_{k}$-th spatial stream, while $\boldsymbol{C}=\underbrace{\left[\begin{array}[]{llll}1&1&\cdots&1\end{array}\right]}_{\lambda_{k}}$ denotes the vector employed to calculate the sum of $\left[\begin{array}[]{llll}Y_{1}&Y_{2}&\cdots&Y_{\lambda_{K}}\end{array}\right]^{T}$. The channel matrix associated with the $k$-th UE is symbolically represented as $\boldsymbol{H}_{k}[i]=\left[\begin{array}[]{cccc}h_{11}&h_{21}&\cdots&h_{\lambda_{0}1}\\ h_{12}&h_{22}&\cdots&h_{\lambda_{0}2}\\ \cdots&\cdots&\cdots&\cdots\\ h_{1\lambda_{k}}&h_{2\lambda_{k}}&\cdots&h_{\lambda_{0}\lambda_{k}}\end{array}\right]$. Furthermore, we consider that the amount of data in the transmission queue within the $i$-th time slot is denoted as $\boldsymbol{X}_{k}[i]=\left[\begin{array}[]{llll}X_{1}&X_{2}&\cdots&X_{\lambda_{0}}\end{array}\right]^{T}$, while the received data is denoted as $\boldsymbol{Y}_{k}[i]=\left[\begin{array}[]{llll}Y_{1}&Y_{2}&\cdots&Y_{\lambda_{K}}\end{array}\right]^{T}$. $D_{k}[i]$ represents the total data volume of the $k$-th user during the $i$-th time slot, the reality of throughput can be formally expressed as follows:

Therefore, the actual throughput of overall system $\mathcal{T}$ during the $T$ period can be expressed as:

However, the inherent instability of wireless systems caused by factors such as system congestion, access contention, and channel state fluctuations, the actual system throughput and Shannon limit within the $i$-th time slot have the following constraints, which are denoted as:

In the context of AI-enabled future wireless communication systems, it is crucial to establish an AI framework that covers user needs and can intelligently optimize communication and computing resource allocation. To accomplish this, we introduce a binary indicator known as $\delta_{k}=\{0,1\}$. Here, $\delta_{k}=0$ denotes the privacy-insensitive user category, while $\delta_{k}=1$ corresponds to the privacy-sensitive user category. It is imperative to address privacy and security concerns during the design phase of this model. Consequently, the computing architecture model, rooted in privacy and security considerations within this system, primarily focuses on adapting the following modules.

1) Privacy-insensitive computing model: For $\delta_{k}=0$, the user type is categorized as privacy-insensitive. Consequently, as there are no privacy security concerns for this type of user, they have an option to select either a semi-centralized or decentralized training mode based on the current distributed learning paradigm. In this mode, the computing model takes into account the energy consumption associated with local actor NN training, server-side critic NN training energy consumption, task calculation energy consumption, as well as the influence of uploading and offloading training parameters on system performance. Therefore, we establish the initial CPU frequency of the $k$-th local UE as $f_{k}$, define the data size of each returned gradient vector (or environment observation space) in the UE as $\zeta_{k}$, and quantify the local computing task as $d_{k}$. In line with the network training process, we denote the number of sampling samples for the $i$-th round of batch gradient descent as $\varepsilon_{k}[i]$, the number of cycles required to process 1 bit of data as $r_{k}$, and the computed value of floating point operations per second (FLOPS) in a single cycle as $c$. Consequently, the energy consumption [7] of the actor NN local calculation energy consumption in the $i$-th time slot is expressed as:

where $\kappa_{k}$ is the CPU effective switched capacitance. In addition, the energy consumption [11] of the actor NN training model is typically denoted as:

Furthermore, the system must also consider the energy consumption associated with computing by the critic NN on the server, which is specifically expressed as:

Similarly, the training energy overhead of the critic NN can be expressed as:

where the size of each channel state sample, representing the environmental observation space in $k$-th UE, is defined as $\rho_{k}$. We denote the number of sampling samples for the $i$-th round of batch gradient descent as $\varepsilon_{k}[i]$. Additionally, computational tasks are quantified as $d_{s}$, while the CPU frequency of the server is denoted as $f_{s}$. Furthermore, we represent the number of cycles required to process 1-bit data as $r_{s}$.

2) Privacy-sensitive computing model: When the type $\delta_{k}=1$, the user type is categorized as privacy-sensitive. Privacy-sensitive users must carefully consider data security concerns, thus need to deploy NNs locally and the adoption of a decentralized distributed training architecture. This training architecture primarily addresses the computational energy consumption of the local NN network, as well as the performance implications of uploading and downloading NN models on the system. Then the energy consumption in the $i$-th time slot of NN local computing is expressed as:

The size of each return gradient in $k$-th agent, which belongs to an NN, is defined as $\zeta_{k}$. Furthermore, $\rho_{k}$ represents the size of the channel state in the wireless system. In terms of the network training process, $\varepsilon_{k}[i]$ represents the number of sampling samples utilized for batch gradient descent in the $i$-th round. Additionally, $r_{k}$ denotes the number of cycles required to process 1-bit. The local NN model training energy consumption is denoted as :

Clearly, based on the aforementioned computational model, the computation cost of the entire distributed system, which includes both privacy-sensitive and non-sensitive users, within the framework of the distributed learning architecture during the $i$-th time slot, can be expressed as:

where $E[{i}]$ refers to the energy consumption of the $i$-th time slot in the overall distributed wireless system.

Without loss of generality, to realize the MAFWDRL application, this paper constructs a wireless communication environment following Wi-Fi standards using the NS3-simulator. It is worth mentioning that the actions and states of individual agents can be adjusted as necessary to suit Non-Wi-Fi standard wireless scenarios. The objective of this paper is to optimize communication and computing resource allocation in scenarios with varying degrees of communication sensitivity, aiming to minimize the dispersion of task weights by federated learning in the wireless system, minimize federated energy consumption or calculation latency, and maximize system throughput. The objective function is denoted as $\mathcal{O}$. The communication network environment is influenced by factors such as aggregation frame length, contention window, and CPU frequency. In this environment, we define the decision variables as follows: $\mathcal{CW}=\left\{{\varepsilon}_{{k}},\forall k\right\}$,$\mathcal{F}\mathcal{L}=\left\{{\xi}_{k},\forall k\right\}$,$\boldsymbol{F}=\left\{{f}_{k},{f}_{s},\forall k,\forall s\right\}$. The optimization problem of the system for the $i$-th training round is formulated as:

where $\mathcal{T}[i]$ denotes system throughput in $i$-th time slot, and calculation energy consumption is denoted as $E[i]$. Constraints (24a) and (24b) define the limits on CPU frequency for STAs and AP, while constraints (24c) and (24d) specify the limits on CW and aggregate frame length in the action space for channel status. Additionally, constraint (24e) presents the maximum queue length limit for channel transmission. Constraints (24f) and (24g), respectively, guarantee the maximum computation delay for privacy-insensitive and privacy-sensitive tasks, thereby ensuring timely system responsiveness and preventing task failures or performance degradation caused by exceeding the expected computation time.

We consider that the wireless system is dynamic and intricate, leading to unpredictable communication conditions. To guarantee the robustness of the system, it is crucial to ensure its adaptability. Evidently, the interdependence among the variables $\mathcal{CW}$, $\mathcal{FL}$, and $\boldsymbol{F}$ renders the objective function $\mathcal{O}$ as a non-convex optimization problem for the wireless system. Consequently, we employ the MAFWDRL method to acquire the optimal action strategy for the system, thereby enabling the objective function $\mathcal{O}$ to obtain an optimal action at any given point in time.

The discrepancy in channel states between different UEs is observed in wireless communication systems, which are influenced by multiple factors, including noise, interference, communication distance, and resource allocation. The performance of FL is affected by the data state distribution (IID and Non-IID) [12, 13, 14]. Based on this, we propose to assess the quality of channels by defining an abstract reward for the system, $R=\boldsymbol{CHX}$.

We consider the communication states between each training round to be mutually independent. The channel state for the $k$-th agent during the training process is denoted as $\boldsymbol{H}_{k}$, and the probability of the state occurring is denoted as $P\left\{X=\boldsymbol{H}_{{k}}\right\}$. The local reward of $k$-th agent in the wireless system is $R_{k}=\boldsymbol{CH_{k}X_{k}}$. The state of the global channel characteristic is denoted as $\boldsymbol{H}_{g}$, and the probability of the state occurring is $P\left\{Y=\boldsymbol{H}_{{g}}\right\}$. Consequently, the global state reward in the wireless system is $R_{g}=\boldsymbol{CH_{g}X_{g}}$. Therefore, the distributed learning expected reward for the $k$-th agent under the current channel state can be expressed as:

where $\boldsymbol{X}_{k}$ is the amount of data to be transmitted. Similarly, in this case, if centralized learning is adopted, the expected reward that should be pursued by the $k$-th agent is denoted as:

where $\boldsymbol{X}_{g}$ denotes the aggregate data volume intended for global transmission, and $M+N$ represents the total number of UEs. The deviation between local and global states is likely to result in a decrease in model accuracy as a consequence of NN model parameter $\theta$. Within each training round, the impact of this factor on the accuracy of the neural network model can be expressed as follows:

where $G(\theta,\boldsymbol{H}\boldsymbol{X})$ represents the model gradient of the current training round, and $\|\boldsymbol{\cdot}\|_{1}$ is 1-norm. Based on the corollary in [6], the difference between the local and global system states can be captured by establishing the correlation between federation weight and loss accuracy. Consequently, based on reference [6, Corollary 1], the model accuracy loss caused by the discrepancy of channel states can be refined as follows:

where $\tau=\frac{\eta\left[1-\left(\sqrt{1+\eta^{2}\beta^{2}-2\eta\lambda}\right)^{\varepsilon}\right]}{1-\sqrt{1+\eta^{2}\beta^{2}-2\eta\lambda}}$, $\eta$ is the learning rate, $\beta$ and $\lambda$ are the derivable constraint coefficients, representing the Lipschitz and smooth assumptions respectively. Modifying the weight parameter $w_{k}$ can alleviate the discrepancy between the local and global states, which can reduce the upper bound of the model accuracy loss $\mathcal{Z}$. Hence, to mitigate the impact of this discrepancy, we define the weight of FedWgt as follows [6, eq.(17)]:

where $w_{k}$ is the FedWgt coefficient.

Introducing exploration noise is an effective approach to address the limited exploration capability in off-policy DRL. During the learning process of NN, our objective is to encourage extensive exploration of the environment by the early-stage network, enabling the collection of a diverse range of environmental states. With the exploration progressing, the scale of noise decreases at an accelerated rate. As the later-stage NN network approaches convergence, the noise level tends to a lower bound for the exploration. Thus, it is essential to design a noise function $f(x)$ that ensures the rate of change increases gradually. This strategy facilitates reasonable exploration and aligns better with the NN learning characteristics. As a result, several conditions need to be met in the design of the noise function $f(x)$.

1) For a function defined on set $B$, if for any points $x_{0}$, $x_{1}$, and $x_{2}$ on the function where $x_{0}<x_{1}<x_{2}$ hold, then the condition

is always satisfied.

2) The noise decay curve defined on set $B$ is monotonically decreasing and the rate of decrease is monotonically increasing. Within the domain of $B$, it exhibits a concave function with gradually increasing and monotonically decreasing rate of change, expressed as:

From a mathematical perspective, a multi-agent task can be defined as a decentralized partially observable Markov decision process (Dec-POMDP) [15, 16, 17], where each agent independently selects actions to react to the environment ultimately. The Markov game of $N+M$ agents can be defined as the set of states $\mathcal{S}$, including the set of action spaces $\mathcal{A}=\left\{a_{1},a_{2},\cdots,a_{N+M}\right\}$, and the set of observation spaces $O=\left\{o_{1},o_{2},\cdots,o_{N+M}\right\}$, where each agent selects actions based on the policy $\pi\left(a_{k}\mid o_{k};\theta_{k}\right)\rightarrow a_{k}$. In addition, this paper considers the issue of local training discrepancies between $o_{1},o_{2},\cdots,o_{N+M}$, and $\mathcal{S}$, and introduces FedWgt $\mathcal{W}=\left\{w_{1},w_{2},\cdots,w_{N+M}\right\}$ to integrate the distributed NN models. Thus, the multi-agent policy can be adjusted as follows: $\pi\left(\tilde{a}_{k}\mid o_{k};w_{k},\theta_{k}\right)\rightarrow\tilde{a}_{k}$, where $\tilde{\mathcal{A}}=\left\{\tilde{a}_{1},\tilde{a}_{2},\cdots,\tilde{a}_{N+M}\right\}$. The state transition equation $\mathcal{E}$ for the entire environment can be represented as: $\mathcal{S}\times\tilde{\mathcal{A}}\rightarrow\mathcal{S}$. At this point, the mapping between actions and states of the $i$-th agent under the current conditions can be directly characterized by rewards: $o_{k}\times\tilde{a}_{k}\rightarrow r_{k}$, where $\mathcal{R}=\left\{r_{1},r_{2},\cdots,r_{N+M}\right\}$. The goal of each agent is to maximize long-term expected returns, i.e., $\mathcal{R}_{k}=\sum_{t=0}^{T}\gamma^{t}r_{k}^{t}$, where $\gamma$ is the discount factor, and $T$ is a complete wireless task period in the wireless system. The Dec-POMDP problem in this paper, specifically in the context of distributed wireless systems, can be represented as a tuple $\mathcal{G}=(\mathcal{S},\tilde{\mathcal{A}},\mathcal{R},\mathcal{E},O,\mathcal{W},\gamma)$.

1) Action Space $\mathcal{A}$: In this paper, the optimization of MAC and CPU parameters to optimize the access and computation efficiency of the wireless channel is considered. The optimization is performed using an NN model, which takes into account the learned channel feature $\boldsymbol{H}$ and the state of the task volume $\boldsymbol{X}$. As a result, the action of agent $k\in\left\{1,2,\cdots,N+M\right\}$ at slot $t$ is defined as $a_{k}^{t}\in\mathcal{A}\triangleq\left[\mathcal{CW},\mathcal{F}\mathcal{L},\mathcal{F}_{s},\mathcal{F}_{c}\right]$, where $\mathcal{CW}$ represents the contention window, and $\mathcal{F}\mathcal{L}$ refers to the aggregation of frame length. Additionally, $\mathcal{F}_{s}$ and $\mathcal{F}_{c}$ denote the CPU computing frequency of the server and clients, respectively.

2) Observation Space $\mathcal{S}$: The observation space in the time slot $t$ is focused on by each agent $k\in\left\{1,2,\cdots,N+M\right\}$ in order to capture the changing environmental states within the system. This observation space enables the direct reflection of the communication state information of the wireless system. The information is primarily composed of the following three parts: 1) SNR of each agent is considered (ignore the interference). The SNR is influenced by factors such as user movement and environmental noise, thereby allowing for the intuitive depiction of the system’s communication quality. 2) The packet loss rate of the system is examined. With the number of ACKs received by the device associated with agent $k$ ($i.e.$ $c_{k}$) and the number of packets sent by agent $k$ ($i.e.$ $d_{k}$) within the time slot $t$, the packet loss rate can be calculated as $\mathcal{P}_{k}=\frac{d_{k}-c_{k}}{d_{k}}$. This indicator provides insights into the extent of data loss during transmission. 3) The percentage of idle time is taken into account. The proportion of data transmission time during the TXOP period by agent $k$ ($i.e.$ $t_{k}$), denoted as $v_{k}$, is monitored. Utilizing the formula $\mathcal{L}_{k}=\frac{t_{i-}v_{k}}{t_{k}}$ allows for the estimation of the proportion of idle time during transmission for agent $k$. Then, the local state of agent $k$ at slot $t$ is denoted as:

At the present moment, the observation space of all agents is represented as $O=\left[o_{1}^{t},o_{2}^{t},\cdots,o_{N+M}^{t}\right]$. Furthermore, the global state information, in the context of the off-policy, can be denoted as $\mathcal{S}=\left[\mathcal{A},O\right]$, where $\mathcal{A}$ represents the action space of all agents.

3) Reward $\mathcal{R}$: The optimization objective function is defined as $\mathcal{O}=\frac{\mathcal{T}}{E}$, where $\mathcal{O}$ is strongly related to the reward, serving as a positive indicator for evaluating network performance. Furthermore, when the system exceeds the defined maximum permissible computing latency threshold, a penalty must be imposed against the DRL model. This penalty is realized by adjusting the reward to a negative value (i.e., $r=-T_{k}$). Such an adjustment aims to instruct the DRL model in recognizing and quantifying the adverse effects that incorrect actions can have on the system’s performance. Consequently, it is inferred that a larger delay signifies a greater deviation of the network from the correct direction. When prioritizing energy consumption, the system demands lower energy consumption within the computing delay limit to achieve an improved overall system reward. In addition, in order to eliminate the impact of numerical deviations of $T$, $\mathcal{T}$ and $E$ on the DRL model as much as possible, we use the arctangent normalization function (i.e., $z(x)=\frac{2}{\pi}\arctan(x)$) to normalize the reward $r$. Given these considerations, the rewards for wireless systems are defined as follows:

where $\mathcal{T}_{k}$, $T_{k}$, and ${E}_{k}$ represent the throughput, computing latency, and computing energy consumption of the $k$-th STA, respectively.

The objective of our distributed wireless system is to optimize the throughput and calculation latency or calculation energy consumption of the overall system. If the actions are entirely independent, certain constraints will arise that impact the system as a whole. Taking these considerations into account, we propose a system optimization framework based on the MAFWDRL method for wireless systems with privacy-sensitive and privacy-insensitive requirements. We consider soft MADRL under the off-policy mechanism as the core optimization algorithm. Our algorithm, as shown in Algorithm 1, consists of three main components: 1) Privacy-sensitive users learn actor NN and critic NN based solely on local information. 2) Privacy-insensitive users, having no concerns about privacy issues, would benefit more from prioritizing the semi-centralized training mode to enhance system performance. 3) A FedWgt learning approach is employed to optimize the global NN model.

1) Fully decentralized MADRL algorithm for privacy-sensitive: Due to data security constraints and the privacy concerns of sensitive users (represented by the set $\boldsymbol{J}\in\left\{j_{1},j_{2},\cdots,j_{{M}}\right\}$, where the identifier $\delta=1$), a fully decentralized soft MADRL optimization algorithm is considered more suitable for communication applications. In this context, the Critic network parameters of ${M}$ agents are denoted as $\boldsymbol{\theta}^{c}=\left\{\vartheta_{j_{1}}^{c},\vartheta_{j_{2}}^{c},\cdots,\vartheta_{j_{{M}}}^{c}\right\}$, and the policy network parameter as $\boldsymbol{\theta}^{p}=\left\{\vartheta_{j_{1}}^{p},\vartheta_{j_{2}}^{p},\cdots,\vartheta_{j_{{M}}}^{p}\right\}$. The action is governed by the policy function $\boldsymbol{u}\left(\boldsymbol{o}\mid\boldsymbol{\theta}^{p}\right)=\left\{u_{j_{1}}\left(o_{j_{1}}\mid\vartheta_{j_{1}}^{p}\right),u_{j_{2}}\left(o_{j_{2}}\mid\vartheta_{j_{2}}^{p}\right),\cdots,u_{j_{{M}}}\left(o_{j_{{M}}}\mid\vartheta_{j_{{M}}}^{p}\right)\right\}$. The updated formula for the training of each critic is as follows:

where $i$ represents the $i$-th time slot, $\mathbb{S}$ represents mini-batch of samples, and the target network parameter denoted as $\widetilde{\vartheta}$, is subject to updates according to the following equation:

where $\varphi$ represents the learning coefficient. Similarly, the updated formula based on Q-value of agent $j_{k}$ for the training of each actor is as follows:

where $\nabla_{\vartheta^{p}_{j_{k}}}\mathcal{J}$ is the policy gradient of the NN model for sensitive users.

2) Semi-centralized MADRL algorithm for privacy-insensitive: For ${N}$ privacy-insensitive users $\boldsymbol{L}\in\left\{l_{1},l_{2},\cdots,l_{{N}}\right\}$, with the identifier $\delta=0$, regardless of data security constraints, sufficient state information can be provided for Critic training. Thus, a model framework featuring centralized training and distributed execution is more conducive to the learning of the system environment by the privacy-insensitive users (i.e., semi-centralized MADRL). The observation space of the $k$-th privacy-insensitive agent, denoted as $l_{k}$, is represented as $\boldsymbol{O}_{l_{k}}=\left(o_{l_{1}},o_{l_{2}},\cdots,o_{l_{{N}}}\right)$. The action of each agent is denoted as $a_{l_{1}},a_{l_{2}},\cdots,a_{l_{{N}}}$, and the policy function follows $\boldsymbol{u}\left(\boldsymbol{o}\mid\boldsymbol{\theta}^{p}\right)=\left\{u_{l_{1}}\left(o_{l_{1}}\mid\vartheta_{l_{1}}^{p}\right),u_{l_{2}}\left(o_{l_{2}}\mid\vartheta_{l_{2}}^{p}\right),\cdots,u_{l_{{N}}}\left(o_{l_{{N}}}\mid\vartheta_{l_{{N}}}^{p}\right)\right\}$. Therefore, the centralized action-value function can be expressed as $\left.Q_{l_{k}}\left(\boldsymbol{O}_{l_{k}};a_{l_{1}},a_{l_{2}},\cdots,a_{l_{N}}\mid\vartheta_{l_{k}}^{c}\right)\right|_{a_{l_{k}}=u\left(o_{l_{c}}\mid\vartheta_{l_{c}}^{p}\right)}$.The updated formula for the training of centralized critic NN is as follows:

The training update formula, within the decentralized architecture, can be expressed as follows:

where $\nabla_{\theta_{l_{k}}}\mathcal{K}$ is the policy gradient of the NN model for insensitive users.

3) Model Ensemble Based on FedWgt: The fully centralized DRL for privacy-sensitive users and the semi-centralized DRL weighted integration strategy for privacy-insensitive users can be expressed as follows:

In order to address the disparity between distributed learning and centralized learning in wireless network optimization, the substitution of the currently learned model parameters $\vartheta_{j_{k}}$ and $\vartheta_{l_{k}}$ with $\Theta_{k}$ is contemplated for different learning architectures. By incorporating $\Theta_{k}$ into the training process, we aim to bridge the gap and compensate for the differences between the two learning paradigms.

The total complexity of the MAFWDRL algorithm is primarily determined by several key factors: the number of agents, the complexity involved in the NN’s forward and backward propagation, the complexity associated with model integration, the interactions between the NN and wireless systems, and the overall number of iterations. The complexity arising from the training process is influenced by the type of training mode employed. Specifically, for $M$ privacy-sensitive agents, the training complexity is given by $\mathrm{O}\left(4M\times\sum_{p=1}^{P-1}B_{p}B_{p+1}\right)$, whereas for $N$ privacy-insensitive agents, it stands at $\mathrm{O}\left(2(N+1)\times\sum_{p=1}^{P-1}B_{p}B_{p+1}\right)$. Here, $B_{p}$ denotes the number of neurons in the $p$-th layer of the network, and $P$ represents the total number of layers in the NN. Furthermore, both the complexities of model integration and agent interactions exhibit a linear relationship with the number of iterations $\mathcal{K}$, represented as $O(\mathcal{K})$. Consequently, the overall complexity of the MAFWDRL algorithm can be accurately quantified as $\mathrm{O}\left((4M+2N+2)\mathcal{K}\sum_{p=1}^{P-1}B_{p}B_{p+1}+\mathcal{K}\right)$.

In this section, our primary objective is to conduct a comprehensive and equitable assessment of noise adjustment’s impact on the performance of distributed wireless systems. To achieve this, we have taken several measures. First, we have categorized all experimental users within the communication scenario as privacy non-sensitive, ensuring a level playing field for our evaluations. Additionally, we have employed the off-policy DRL algorithm MADDPG for our testing procedures.

To facilitate meaningful comparisons, we have considered two distinct noise adjustment functions. The first is the conventional noise linear decrease function, denoted as $f(t)=-\phi t+n_{0}$. The second is an illustrative example, which adheres to the criteria outlined in Section III, Subsection B, expressed as $g(t)=-(\eta t)^{3}+n_{0}$. Here, it is important to note that $n_{0}$ serves as a controlling factor, determining the maximum offset scale for actions that MADDPG can select. Meanwhile, $\phi$ and $\eta$ play crucial roles in regulating the magnitude of decrease in each iteration for the linear function $f(t)$ and the concave function $g(t)$, respectively. Our criteria for assessing system performance depend on the observed throughput of the entire system, which provides an objective basis for evaluating the various noise adjustment approaches.