DRL-based Distributed Resource Allocation for Edge Computing in Cell-Free Massive MIMO Network

We consider a cell-free massive MIMO-enabled mobile edge network, depicted in Fig. 1, where a central processing unit (CPU) with an edge server of finite capacity ${f^{CPU}}$ (in cycles per second) is connected to $M$ single-antenna access points (APs) via ideal fronthaul links. Furthermore, $M$ APs serve $K$ single-antenna users, such that $M\gg K$, entailing to the widely known system model for cell-free massive MIMO [5]. Let ${{\cal M}}=\left\{{1,2,...,M}\right\}$ and ${{\cal K}}=\left\{{1,2,...,K}\right\}$ denote sets of AP and user indices, respectively. Besides, each user $k\in{{\cal K}}$ is also equipped with limited local processor with maximum capability $f_{k}^{\max}$ (in cycle per second).

Let the channel between $m$-th AP and $k$-th user is given as ${{g}_{mk}}=\beta_{mk}^{{}^{1}/{}_{2}}{{h}_{mk}}$ where ${{\beta}_{mk}}$ is a large-scale channel gain coefficient that models the effects of path loss and shadowing while ${h_{mk}}\sim{{\cal C}{\cal N}}\left({0,1}\right)$ represents small-scale channel fading. Let ${{\tau}_{c}}$ denote a channel coherence period in which ${{h}_{mk}}$ remains the same. The coherence period ${{\tau}_{c}}$ is divided into pilot transmission interval of ${{\tau}_{p}}$ samples and uplink data transmission interval of (${{\tau}_{c}}-{{\tau}_{p}}$) samples for offloading computation to the edge server in the CPU.

At the beginning of each coherence time, each user $k\in{{\cal K}}$ simultaneously transmit a pilot sequence ${{\mathbf{\psi}}_{k}}\in{{\mathsf{\mathbb{C}}}^{{{\tau}_{p}}\times\,1}}$, such that ${{\left\|{{\mathbf{\psi}}_{k}}\right\|}^{2}}=1$, which is used to estimate the respective channels ${{\hat{g}}_{mk}}$ at each AP $m\in{{\cal M}}$. Assuming pairwise orthogonal sequences $\left\{{{\mathbf{\psi}}_{1}},{{\mathbf{\psi}}_{2}},.....,{{\mathbf{\psi}}_{K}}\right\}$, i.e., ${{\tau}_{p}}=K$, we ignore pilot contamination problem in the current model. Then, the received pilot vector at the $m$-th AP, $\mathbf{y}_{m}^{p}\in{{\mathsf{\mathbb{C}}}^{{{\tau}_{p}}\times\,1}}$, can be represented as

where $q_{k}^{p}$ is the pilot transmit power, and $\mathbf{\omega}_{m}^{p}$ is a $\,{{\tau}_{p}}$-dimensional additive noise vector with independent and identically distributed entries of ${\bf{\omega}}_{m}^{p}\sim{{\cal C}{\cal N}}\left({0,\sigma_{m}^{2}}\right)$. Based on the received vector $\mathbf{y}_{m}^{p}$, the least-square (LS) channel estimate ${{\hat{g}}_{mk}}$ can be expressed as:

The channel estimates are used to decode the uplink transmitted data of the users. After pilot transmission, the users transmit offloaded data to the APs. Let ${{x}_{k}}$ denote the uplink data of user $k$, with transmit power of ${{p}_{k}}$, which is determined as ${{p}_{k}}={{\eta}_{k}}\,p_{k}^{\max}$, where ${{\eta}_{k}}$, and $p_{k}^{\max}$ represent power control coefficient and maximum uplink power of the $k$-th user, respectively. Then, the received signal $y_{m}^{u}$ at the $m$-th AP is represented as

It is important to ensure the scalability of cell-free massive MIMO system, in terms of complexity for pilot detection and data processing. To this end, we limit the number of APs that serve the $k$-th user to ${N_{k}}<M$, and form a user-centric cluster of APs ${{{\cal C}}_{k}}\subset\left\{{A{P_{1}},A{P_{2}},\cdots,A{P_{N_{k}}}}\right\}$, by including all APs with the largest ${{\beta}_{mk}}$ to ${{{\cal C}}_{k}}$ until the maximum allowable cluster size is reached, i.e., ${N_{k}}={C^{\max}}$. Thus, the transmitted data by user $k$ is only decoded by the APs in ${{{\cal C}}_{k}}$. Then, after performing maximum ratio combining (MRC) at each AP $m\in{{{\cal C}}_{k}}$ the quantity $\hat{g}_{mk}^{*}y_{m}^{u}$ is transmitted to the CPU via a fronthaul link. The received soft estimates are combined at the CPU to decode the data transmitted by user $k$ as follows:

Then, the uplink signal-to interference and noise ratio (SINR) ${\gamma_{k}}$ for user $k$ can be expressed as:

Given the system bandwidth $W$, the uplink rate of user $k$ is given as ${R_{k}}=W{\log_{2}}\left({1+{\gamma_{k}}}\right)$ by Shannon’s theory.

We assume each user $k$ has a computationally intensive task of ${{\cal T}_{k}}\left(t\right)$ bits at the beginning of every discrete time step $t$, whose duration $\Delta t={{\tau}_{c}}$. The task sizes in different time steps are independent and uniformly distributed over $\left[{{\cal T}_{\min,}}\,{{\cal T}_{\max,}}\right]$, for every user $k\in{{\cal K}}$. Let $t_{k}^{d}$ denote the maximum tolerable delay of user $k$ to complete execution of the given task. Considering the delay constraint and energy consumption, the $k$-th user processes ${\cal T}_{k}^{local}\left(t\right)$ bits locally while offloading the remaining ${\cal T}_{k}^{offload}\left(t\right)$ bits to the edge server at the CPU.

According to the aforementioned communication and parallel computation models, we intend to jointly optimize the local processor speed $f_{k}^{local}\left(t\right)$ and uplink transmission power ${p_{k}\left(t\right)}$ for every user $k\in{{\cal K}}$ at each time step $t$, in order to minimize the total energy consumption of all users subject to the respective user-specific delay requirements. The JCCRA problem can mathematically be formulated to jointly determine $\left({{\alpha_{k}}\left(t\right),{\eta_{k}}\left(t\right)}\right),\forall k\in{{\cal K}}$ as follows:

(12) is a stochastic optimization problem in which the objective function and the first condition involve constantly changing random variables, i.e., task size and channel conditions. Consequently, the problem must be solved frequently, i.e., at every time step $t$, and the solution must converge rapidly within the ultra-low delay constraint. Therefore, it is challenging to solve the problem using iterative optimization algorithms with reasonable complexity. Furthermore, relying on a one-shot optimization, it is difficult to guarantee a steady long-term performance with conventional algorithms since they cannot adapt to the changes in the mobile edge network.

Each user is implemented as a DRL agent that learn to map local observation of the environment to efficient JCCRA actions by sequentially interacting with the mobile edge network in discrete time steps. Let ${{{\cal O}}_{k}}$, and ${{\cal S}}$ denote the local observation space of agent $k\in{{\cal K}}$, and the complete environment state space, respectively. At the beginning of each step $t$, agent $k$ perceives the local observation of the environment state ${o_{k}}\left(t\right):{{\cal S}}\mapsto{{{\cal O}}_{k}}$, and takes an action ${a_{k}}\left(t\right)\in{{{\cal A}}_{k}}$ from its action space ${{{\cal A}}_{k}}$ according to the current JCCRA policy ${{\mu}_{k}}$. The interaction with the environment produces the next observation of the agent ${o_{k}}\left({t+1}\right)\in{{{\cal O}}_{k}}$ and a real-valued scalar reward ${r_{k}}\left(t\right):{{\cal S}}\times{{{\cal A}}_{k}}\mapsto\mathsf{\mathbb{R}}$ that guides the agent to constantly improve the policy until it converges. The goal of the agent is, therefore, to derive an optimal JCCRA policy $\mu_{k}^{*}$ that maximizes the expected long-term discounted cumulative reward, defined as ${{J}_{k}}\left({{\mu}_{k}}\right)=\mathbb{E}\left[\sum\limits_{t=1}^{T}{{{\varepsilon}^{t-1}}{{r}_{k}}}\left(t\right)\right]$, where $\varepsilon\in\left[0,\,1\right]$ is a discount factor and $T$ is the total number of time steps (horizon). In the sequel, we define the local observation, action, and reward of agent $k$ for the JCRRA at a given time $t$.

During the offline training phase, the agents can be trained in a central unit, which presents an opportunity to share extra information among the agents for easing and accelerating the training process. Specifically, agent $k$ has now access for the joint action $a\left(t\right)=\left({{a}_{1}}\left(t\right),...,{{a}_{K}}\left(t\right)\right)$, and full observation of the environment state ${s_{k}}\left(t\right)=\left({{o_{k}}\left(t\right),{o_{-k}}\left(t\right)}\right)$, where ${o_{-k}}\left(t\right)$ is the local observation of other agents at time step $t$. The extra information endowed to the agents, however, is discarded during the execution phase, meaning the agents rely on their local observation entirely to make JCCRA decisions in a fully distributed manner.

As an extension of the single-agent deep deterministic policy gradient (DDPG) algorithm [9], the MADDPG is an actor-critic policy gradient algorithm. Specifically, the MADDPG agent $k$ employs two main neural networks: actor network with parameters $\theta_{k}^{\mu}$ to approximate a JCCRA policy ${{\mu}_{k}}\left({{o}_{k}}|\theta_{k}^{\mu}\right)$ and a critic network, parametrized by $\theta_{k}^{Q}$, to approximate a state-value function ${{Q}_{k}}\left({{s}_{k}},a|\theta_{k}^{Q}\right)$, along with their respective time-delayed copies, $\theta_{k}^{{{\mu}^{\prime}}}$ and $\theta_{k}^{{{\mu}^{\prime}}}$, which serve as targets. At time step $t$, the actor deterministically maps local observation ${{o}_{k}}\left(t\right)$ to a specific continuous action ${{\mu}_{k}}\left({{o}_{k}}\left(t\right)|\theta_{k}^{\mu}\right)$ and then, a random noise process ${{{\cal N}}_{k}}$ is added to generate an exploratory policy such that ${{a}_{k}}\left(t\right)={{\mu}_{k}}\left({{o}_{k}}\left(t\right)|\theta_{k}^{\mu}\right)+{{{\cal N}}_{k}}\left(t\right)$. The environment, which is shared among the agents, collects the joint action $a\left(t\right)=\left\{{{a_{k}}\left(t\right),\forall k\in{{\cal K}}}\right\}$ and returns the immediate reward ${{r}_{k}}\left(t\right)$ and the next observation ${{o}_{k}}\left(t+1\right)$ to the respective agents. To make use of the experience in later decision-making steps efficiently and improve the stability of the training, the agent’s transition along with the extra information ${{e}_{k}}\left(t\right)=\left({{s}_{k}}\left(t\right),a\left(t\right),{{r}_{k}}\left(t\right),{{s}_{k}}\left(t+1\right)\right)$ is saved in the replay buffer ${\cal D}_{k}$ of the agent.

To train the main networks, a mini batch of $B$ samples $\left.{\left({s_{k}^{i},a^{i},r_{k}^{i},s_{k}^{i+1}}\right)}\right|_{i=1}^{B}$, is randomly drawn from the replay buffer ${\cal D}_{k}$, where $i$ denotes a sample index. The critic network is updated to minimize the following loss function:

where $y_{k}^{i}$ is the target value expressed as $y_{k}^{i}={{\left.r_{k}^{i}+\varepsilon\,\,{{{{Q}^{\prime}}}_{k}}\left(s_{k}^{i+1},{{a}_{i+1}}|\theta_{k}^{Q^{\prime}}\right)\right|}_{{{a}_{{}^{i+1}}}=\left\{{{{{\mu}^{\prime}}}_{k}}\left(o_{k}^{i+1}\right),\forall k\in\cal{K}\right\}}}$. On the other hand, the parameters of the actor network are updated along the deterministic policy gradient given as

The target parameters in both actor and critic networks are updated as $\theta_{k}^{\mu^{\prime}}\leftarrow\tau\theta_{k}^{\mu}+\left({1-\tau}\right)\theta_{k}^{\mu^{\prime}}$, and $\theta_{k}^{Q^{\prime}}\leftarrow\tau\theta_{k}^{Q}+\left({1-\tau}\right)\theta_{k}^{Q^{\prime}}$, respectively, where $\tau$ is a constant close to zero.

It is worth mentioning that the proposed JCCRA scheme is trained offline. Once it converges, real-time decision making can be conducted in a fully distributed fashion by simply performing inference with the trained actor network of each agent, relying on the respective local observation only.