Intelligent Reflecting Surface Configurations for Smart Radio Using Deep Reinforcement Learning

For IRS assisted wireless communications, the channel model between BS and a certain user $k$ can be represented as

where the user $k$ is equipped with a single antenna, $\mathbf{h}_{BU_{k}}\in\mathbb{C}^{N_{B}\times 1}$ is the channel response vector between the user $k$ and BS, $\mathbf{H}_{BR}\in\mathbb{C}^{N_{B}\times N_{R}}$ is the channel response matrix between BS and IRS, $\mathbf{h}_{RU_{k}}\in\mathbb{C}^{N_{R}\times 1}$ is the channel response vector between the user $k$ and IRS, $\bm{\Theta}=\operatorname{diag}\{\bm{\theta}\}$, and $\bm{\theta}\in\mathbb{C}^{N_{R}\times 1}$ (with $|\bm{\theta}(n)|=1$) is the phase shift vector of the IRS. Accordingly, the multi-user channel is written as

where

And the relationship between the aggregated equivalent channel $\mathbf{H}$ and the sub-channels is shown in Figure 1.

The objective of reinforcement learning based IRS configuration is to develop a widely compatible method that can be deployed in various scenarios of wireless communications without any knowledge of the wireless system’s internal working mechanism. Mathematically, the problem is formulated as

where $P_{m}$ is the performance metric of the wireless system that is to be optimized. $P_{m}$ is dependent on the wireless channel $\mathbf{H}$, and $\mathbf{H}$ is dependent on the reflection pattern $\bm{\theta}$. $\bm{\varphi}[n]$ is the quantized phase selected from a finite set $\mathcal{B}=\left\{-\pi,\frac{-2^{r}+2}{2^{r}}\pi,\frac{-2^{r}+4}{2^{r}}\pi,\cdots,\pi\right\}$ with $2^{r}+1$ possible values.

It is worth mentioning that the model-free control does not need to know the exact relationship between the objective $P_{m}$ and variable $\bm{\theta}$.

Without loss of generality, we use a typical scenario in wireless communication, i.e., TDD multi-user MIMO, to illustrate our design philosophy. In TDD, by exploiting the channel reciprocity, the BS can estimate the downlink channel from the pilot of the uplink channel. Thus, TDD multi-user MIMO consists of two stages (refer to Figure 2), i.e., uplink pilot transmission and downlink data transmission[30, 31].

At uplink stage, the pilot transmits from multiple users to BS simultaneously. The received pilot signal is represented as

where $\mathbf{S}\in\mathbb{C}^{K\times K}$ is the pilot pattern, $\mathbf{N}\in\mathbb{C}^{N_{B}\times K}$ is the additive white Gaussian noise. Upon receiving the pilot, BS performs minimum mean square error (MMSE) estimation of the channel matrix, i.e.,

When $\mathbf{S}$ is an unitary matrix, (5) is further expressed as

At downlink stage, data transmission with zero-forcing (ZF) precoding is performed, and the precoding matrix is represented as

where $\mathbf{D}_{p}={\rm diag}([\frac{1}{\|\mathbf{m}_{1}\|_{2}},\frac{1}{\|\mathbf{m}_{2}\|_{2}},\cdots,\frac{1}{\|\mathbf{m}_{K}\|_{2}}])$ is for power normalization. The received signal of user $k$ is given by

where $x_{k}$ is the signal intended to user $k$ ($\mathbb{E}(x_{k})=0$ and $\mathbb{E}(|x_{k}|^{2})=1$, $\forall k\in\{1,\cdots,K\}$), and $n_{k}\sim\mathcal{CN}(0,\sigma_{k}^{2})$ is the additive white Gaussian noise. Thus, the signal-to-noise ratio of the $k$-th user is

For a communication system, the performance metrics can be SINR, data rate, frame error rate (FER), and etc. Without loss of generality, we adopt the sum data rate as the performance metric, i.e.,

We assume the Rician channel model for $\mathbf{h}_{BU_{k}}$, $\mathbf{H}_{BR}$ and $\mathbf{h}_{RU_{k}}$. Take $\mathbf{H}_{BR}$ as an example, it is represented as

where $\mathbf{H}_{BR,LoS}$ denotes the deterministic LoS component, $\mathbf{H}_{BR,NLoS}$ denotes the fast fading NLoS component, and the component of which is independent and identically distributed (i.i.d.) circularly symmetric complex Gaussian random variables with zero-mean and unit variance, and $K$ is the ratio between the power in the LoS path and the power in the NLoS paths [32].

The LoS component is position-dependent and is thus slow-time-varying; The NLoS components are caused by the multi-path effects and are thus fast-time-varying [33]. Combining the characteristics of wireless channel with the setting of reinforcement learning, we introduce the following two concepts.

(1) Channel block: One channel block consists of the uplink pilot transmission stage and downlink data transmission stage (as shown in Figure 2), and the channel matrix is constant during the channel block.

(2) Channel episode: One channel episode consists of $T$ channel blocks (as shown in Figure 2). The LoS component within one channel episode remains constant; The NLoS components change over time, and the NLoS components of different channel blocks are i.i.d.

We aim to achieve stand-alone operation of the wireless communication system and the IRS, and our design includes the following characteristics.

• •

  Wireless Communication System: The wireless communication system is almost unaware of the existence of the IRS, except that it needs to feed back its instantaneous performance to the IRS controller. In this regard, the uplink pilot transmission and downlink data transmission exactly follow the conventional structure in Figure 2.

• •

  IRS: The IRS is strictly regarded as part of the wireless channel and will not be jointly designed with the wireless communication system. The configuration of IRS is based on (a) the performance feedback from the wireless system and (b) its learned policy through trail-and-error interaction with the dynamic environment. And the IRS is unaware of the working mechanism of the wireless communication system.

A salient advantage of the proposed design is that the IRS can be deployed in various wireless communication applications, e.g., Wi-Fi, 4G-LTE, 5G-NR, without updating their existing protocols, which will speed up the roll-out of IRSs. Another benefit is that, by treating the existing wireless communication system as a black box, the configuration of IRS does not require the overhead-demanding channel sounding process to acquire the CSI of the subchannels, i.e., $\mathbf{H}_{BU}$, $\mathbf{H}_{BR}$, and $\mathbf{H}_{RU}$, that constitute the aggregated equivalent channel $\mathbf{H}$.

Our design is primarily based on the reinforcement learning technology. Specifically, the IRS and its controller are the agent, the wireless communication system, which comprises transmitter, wireless channel, and receiver, is the environment. The relationship between the different parties is given in Figure 3. Initially, the agent takes random actions and the environment responds to those actions by giving rise to rewards and presenting new situations to the agent [34, 35]. Through trail-and-error interaction with the wireless communication system, the agent gradually learns the optimal policy to maximize the expected return over time. In this regard, IRS, which is capable of changing the radio environment, is analogous to the human body, and the reinforcement learning method, which guides the action of IRS, is analogous to the human brain. The integration of the IRS and the reinforcement learning method is the pathway to creating the smart radio environment.

To facilitate the presentation of our design, we briefly introduce some key concepts of DRL in this subsection.

To apply reinforcement learning to model-free IRS configuration, we firstly model IRS assisted wireless communications as an MDP.

• •

  Agent: The agent is IRS controller, which is capable of autonomously interacting with the environment via IRS to meet the design objectives.

• •

  Environment: The environment refers to the things that the agent interact with, which includes BS, wireless channel, IRS, and mobile users.

• •

  State: To facilitate the accurate prediction of expected next rewards and next states given an action, we define the state as $\{\mathbf{H},\bm{\theta}\}$, which consists of two sub-states, namely the equivalent wireless channel $\mathbf{H}$ and the reflection vector $\bm{\theta}$ of IRS.

• •

  Action: The action is defined as the incremental phase shift of the current reflection pattern, i.e.,

  $\displaystyle\bm{\theta}^{(t+1)}=\bm{\theta}^{(t)}\odot\Delta\bm{\theta}^{(t)}$

  (25)

  where $\odot$ is the Hadamard (element-wise) product, $\bm{\theta}^{(t)}$ is the reflection pattern at the $t$-th channel block, and $\Delta\bm{\theta}^{(t)}$ is the incremental phase shift of $\bm{\theta}^{(t)}$. We use the subset (or full set) of the discrete Fourier transform (DFT) vectors as the action set. For example, when the size of action space is $5$, we set $\mathcal{A}=\left\{\mathbf{v}(-\frac{6}{N_{R}}),\mathbf{v}(-\frac{2}{N_{R}}),\mathbf{v}(0),\mathbf{v}(\frac{2}{N_{R}}),\mathbf{v}(\frac{6}{N_{R}})\right\}$, where $\mathbf{v}(\Psi_{R})$ is the steering vector ¹¹1Without loss of generality, we assume that the reflector array of IRS is a uniform linear array (ULA)., i.e.,

  $\displaystyle\mathbf{v}(\Psi_{R})$

  $\displaystyle=\left[1,\;e^{j\pi\Psi_{R}},\;\cdots,\;e^{j(N_{R}-1)\pi\Psi_{R}}\right]^{T}$

  When $\Delta\bm{\theta}^{(t)}=\mathbf{v}(0)$, the sub-state $\bm{\theta}$ stays unchanged, and the sub-state $\mathbf{H}$ changes merely due to the variation of NLoS components; $\Delta\bm{\theta}^{(t)}=\mathbf{v}(-\frac{2}{N_{R}})$ and $\Delta\bm{\theta}^{(t)}=\mathbf{v}(\frac{2}{N_{R}})$ are towards the opposite directions, which enables the agent to quickly correct from a negative action; $\Delta\bm{\theta}^{(t)}=\mathbf{v}(-\frac{6}{N_{R}})$ and $\Delta\bm{\theta}^{(t)}=\mathbf{v}(\frac{6}{N_{R}})$ are used to speed up the transition of reflection pattern.

• •

  Reward: The immediate reward after transition from $s$ to $s^{\prime}$ with action $a$ is defined as

  $\displaystyle R=\left\{\begin{array}[]{cc}P_{m},&\;{\rm when}\;\;P_{m}\geq P_{th}\\ P_{m}-100,&\;{\rm when}\;\;P_{m}<P_{th}\end{array}\right.$

  (28)

  where $P_{th}$ is a performance threshold. When $P_{m}$ is less than $P_{th}$, we add an penalty $-100$ to encourage the IRS to maximize performance, while maintaining an acceptable performance above the threshold.

Based on the modeled MDP and the basics of DRL presented in Subsection B, we propose to maximize the expected return (cumulative discounted future reward) using Algorithm 1. Some of the key techniques applied in Algorithm 1 are explained as follows.

In this subsection, we summarize the work flow of the proposed model-free IRS configuration. To this end, we plot the sequence diagram in Figure 5.

According to Figure 5, in a specific loop, the IRS is configured with a reflection pattern ($\bm{\theta}^{(t+1)}=\bm{\theta}^{(t)}\odot\Delta\bm{\theta}^{(t)}$) according to the $\epsilon$-greedy policy, and then UEs and BS perform uplink pilot transmission and downlink data transmission sequentially as if there exists no IRS. After that, BS sends the estimated channel matrix ($\hat{\mathbf{H}}^{(t+1)}$) to IRS controller, which can be fulfilled through wired communications, and UEs send back their performance metrics to the IRS controller. Finally, according to the received channel estimate ($\hat{\mathbf{H}}^{(t+1)}$) and performance feedback ($P_{m}^{t+1}$), IRS controller derives the tuple $\langle\{\hat{\mathbf{H}}^{(t)},\bm{\theta}^{(t)}\},\Delta\bm{\theta}^{(t)},R^{(t+1)},\{\hat{\mathbf{H}}^{(t+1)},\bm{\theta}^{(t+1)}\}\rangle$ and stores it in the FIFO queue as the training data for the DQN $\widetilde{Q}(s_{t},a;\mathbf{w})$.

Compared with the traditional TDD multi-user MIMO, the extra efforts of incorporating IRS are merely the feedback of $\hat{\mathbf{H}}^{(t)}$ and $P_{m}^{t+1}$. The former can be easily achieved via wired communications between BS and IRS, and the latter costs negligible wireless communication resources of the mobile UEs. It is also noteworthy that IRS controller is unaware of the working mechanism of BS and UEs and does not require the CSI of the sub-channels.

ESC is model-free method to realize a learning-based adaptive controller for maximizing/minimizing certain system performance metrics [42, 43]. The first application of ESC can be traced back to the work of the French engineer Leblanc in 1922 to maintain an efficient power transfer for a tram car [44]. The basic idea of ESC is to add a dither signal (e.g., sinusoidal signal [45, 46], and random noise [47]) to the system input and observing its effect on the output to obtain an approximate implicit gradient of a nonlinear static map of the unknown system [48, 42].

According to the design philosophy of ESC, we propose a dither-based model-free control of IRS as in Figure 6. Our design consists of three parts, i.e., dither signal generation module, ascent direction estimation module, and parameter update module. Dither signal generation module generates random dither/pertubation signal to probe the response $P_{m}(\cdot)$ of the system; gradient estimation module determines the update direction of the system input according to the system performance $P_{m}(\bm{\varphi}+\Delta\bm{\varphi})$ to guarantee the monotonic increase of the performance, and it also guides on-off switch of the random dither signal generation; parameter update module updates the system input $\bm{\varphi}$ according to the estimated direction.

Specifically, the iterative process in Figure 6 runs as follows.

Generate a small random dither signal through uniform random distribution²²2The parameter selection will be explained in the following context., i.e.,

Then, add the dither signal $\Delta\bm{\varphi}$ to the parameter $\bm{\varphi}$, use $\bm{\varphi}+\Delta\bm{\varphi}$ as the input of the system, and receive the feedback of the performance metric $P_{m}(\bm{\varphi}+\Delta\bm{\varphi})$.

It is noteworthy that each iteration uses one or two time intervals, and each iteration can guarantee the monotonic increase of the performance metric $P_{m}$, which is validated in the following proposition.

To obtain further insights into the proposed dither-based method, we make a comparison with the well-known iterative algorithm – gradient ascent (descent in minimization problems) search.

For gradient ascent search algorithm, $\bm{\varphi}$ in each iteration is updated as follows.

When the step size $\gamma$ is small, the iteration will almost surely guarantee the increase of $P_{m}(\bm{\varphi})$, because

Recall that the action space for DRL is intentionally restrained for a fast convergence rate, whilst dither-based iterative method relies on the small-scale phase shift. Therefore, dither-based iterative method is complementary to the action in DRL and can be applied to enhance the action of DRL.

The enhanced action of DRL is defined as follows.

To be compatible with the enhanced action, the frame structure needs to be updated as in Figure 7. In the first $K$ time slots, UEs transmit pilots and BS performs channel estimation with the reflection pattern in (43), and in the subsequent $(L-1)K$ time slots, UEs repeatedly transmit pilots and BS performs channel estimation, while the reflection pattern updates as in (44). It is noteworthy that, as the performance feedback is done once per channel block, the performance metric used for the dither-based method is an approximation derived by replacing the authentic channel response $\mathbf{H}$ in (10) with the channel estimate $\hat{\mathbf{H}}$.

The BS is equipped with a ULA that is placed along the direction $[1,0,0]$ (i.e., x-axis), and IRS is a ULA, which is placed along the direction $[0,1,0]$ (i.e., y-axis), UEs are equipped with a single antenna, the user number is $K=2$, BS antenna number is $N_{B}=2$, and IRS reflector number if $N_{R}=32$. The element antennas/reflectors of BS and IRS are both with half wavelength spacing. The position of BS is $[0,0,10]$, the position of IRS is $[-2,5,5]$, and the UEs are uniformly distributed in the area $[0,10)\times[0,10)$ with the height being $1.5$. The noise variance at BS side is $\sigma_{B}^{2}=0.1$, the noise variance at UE side is $\sigma^{2}_{k}=0.5,\;\forall k\in\{1,\cdots,K\}$. Each channel episode consists of $20$ channel blocks. In each channel episode, the LoS component is generated by randomly selecting the user locations within the area $[0,10)\times[0,10)$, and the LoS component is time-invariant within the $20$ channel blocks of that channel episode.

The LoS channel between BS and IRS is

where the steering vectors are represented as

and, according to [49], the directional cosines $\Psi_{R},\Psi_{B}$ are given by

where the direction vector $\mathbf{e}_{BR}$ is determined by the relative position of BS and UE, i.e.,

The NLoS components are Gaussian distributed, i.e., $\mathbf{H}_{BR,NLoS}(\ell,\kappa)\in\mathcal{CN}(0,1)$. The channel matrix $\mathbf{H}_{BU}$ and $\mathbf{H}_{RU}$ are generated in the same way.

In Figure 8, we study the performance of the proposed DRL scheme (with the action set $\mathcal{A}_{5}$) under different values of Rician factor. The x-axis represents the episode, and the y-axis represents the moving average of $P_{m}$ (window length is $64$). As can be seen, the proposed DRL significant outperforms the benchmark schemes, i.e., random reflection and multi-armed bandit (MAB). Different from DRL, the actions of random reflection and MAB that we used are absolute phase shift, and the action set is the DFT vectors. Although all the three schemes are independent of the sub-channel CSI, their utilizations of the other information are different. Random reflection is independent of any information and undoubtedly achieves the worst performance; MAB assumes a fixed distribution of rewards and explores the reward distributions of all arms. However, MAB fails to describe the state of the environment and to build the connection between the action and the environment; DRL defines an appropriate state to represent the agent’s “position” within the environment and learns the quality of a state-action combination using the DQN from the information of rewards and states, which enables the agent to choose the best action to maximize the returns. We can also find that the performance gap of DRL and the benchmarks schemes becomes larger with the increase of Rician factor $K_{Rician}$, which indicates that the effectiveness of DRL is also dependent on the radio environment.

In Figure 9, we study the impacts of action size to the performance of the proposed DRL scheme when the Rician factor is $K=10$. In addition to the action set $\mathcal{A}_{5}=\left\{\mathbf{v}(-\frac{6}{N_{R}}),\mathbf{v}(-\frac{2}{N_{R}}),\mathbf{v}(0),\mathbf{v}(\frac{2}{N_{R}}),\mathbf{v}(\frac{6}{N_{R}})\right\}$ defined in Section III, we adopt the action set $\mathcal{A}_{3}=\left\{\mathbf{v}(-\frac{2}{N_{R}}),\mathbf{v}(0),\mathbf{v}(\frac{2}{N_{R}})\right\}$ and action set $\mathcal{A}_{32}=\left\{\mathbf{v}(-1),\mathbf{v}(-1-\frac{2}{N_{R}}),\cdots,\mathbf{v}(1-\frac{2}{N_{R}})\right\}$ (namely, DFT matrix) as the benchmarks. From Figure 9, we can see that $\mathcal{A}_{5}$ is the fastest to converge, while $\mathcal{A}_{32}$ is the slowest. Although a large action size will speed up the response rate of the agent, it will, on the other hand, demand more time for the DQN to converge. In the convergence region, we find that $\mathcal{A}_{5}$ and $\mathcal{A}_{32}$ achieve the similar performance, while $\mathcal{A}_{3}$’s performance is inferior. It indicates that a well-design action set with a moderate size might be better than the small action size and the over-large action size. In Figure 9, the average sum rate $\mathbb{E}(P_{m})$, and the probability $P_{r}(P_{m}>10)$ are presented in the bar chart. For $\mathcal{A}_{5}$, $\mathbb{E}(P_{m})=9.89$ bps/Hz and $P_{r}(P_{m}>10)=58.05\%$; for $\mathcal{A}_{3}$, $\mathbb{E}(P_{m})=9.34$ bps/Hz and $P_{r}(P_{m}>10)=42.9\%$; for $\mathcal{A}_{32}$, $\mathbb{E}(P_{m})=9.60$ bps/Hz and $P_{r}(P_{m}>10)=49.35\%$. It further verifies the importance of action set design.

In Figure 10, we study the performance of the ESC-inspired dither-based iterative method in a specific channel block. Each time interval of $x$-axis consists of $K$ time slots, which is the pilot length required by the BS to estimate the multi-user channel $\mathbf{H}$. As can be seen that the performance metric $P_{m}$ for dither-based method is almost monotonically increasing over time. Note that the performance metric used to guide the dither-based iterative method is an approximation, rather than the authentic feedback. Thus, the slight fluctuation of the performance curve is reasonable. For the purpose of comparison, we adopt the model-based method as the benchmark, in which the perfect sub-channel CSI, namely, $\mathbf{H}_{BU}$, $\mathbf{H}_{BR}$, and $\mathbf{H}_{RU}$, is available. The optimal reflection coefficient vector $\bm{\theta}$ is derived through solving the optimization problem (3). Recall that the difference from model-free control is that the exact relationship between the objective function $P_{m}$ and the variable $\bm{\theta}$ is known in model-based methods. Due to the discrete nature of the feasible region $\mathcal{B}$, the optimization problem is intractable. Hence, we manage to solve it using the alternating optimization technique, which alternatively freezes $N_{R}-1$ reflection coefficients and optimize only $1$ reflection coefficient. According to Figure 10, the model-free dither-based method achieves almost the same performance as the model-based alternating optimization when the time index is greater than $500$. However, in practice, we have to weigh the cost of time resources against the benefits. As the dither-based method needs to sample $\hat{\mathbf{H}}$, one iteration means the cost of a unit of time resource in wireless communications. Thus, in order to balance the time allocation between pilot transmission and data transmission, the time resources dedicated to dither-based method (i.e., the time for pilot transmission) should be deliberately selected according to channel dynamics (i.e., the length of a channel block).

In Figure 11, we study the performance of the integrated DRL and ESC method (with the action set $\mathcal{A}_{5}$) in different channel dynamics. The normalized $P_{m}$ is the obtained by multiplying $P_{m}$ by the coefficient $\frac{L_{B}-L}{L_{B}}$, where $L_{B}$ is the channel block length and $L$ is the training length. Take $L_{B}=100$ and $L=1$ as an example, the first $L=1$ time interval is used for pilot transmission and the rest $L_{B}-L=99$ time intervals are used for data transmission. It is also noteworthy that when $L=1$ the scheme is DRL only, and when $L\geq 2$ the scheme is the integrated DRL and ESC. In addition, we adopt the action set $\mathcal{A}_{5}$. From the figure, we can see that when the channel block length $L_{B}=100$, DRL outperforms the integrated DRL and ESC with $L=20,40$. However, when the channel block length increases, the integrated DRL and ESC gradually becomes superior, which verifies its effectiveness in the slow fading channel. Therefore, the parameter $L$ of the integrated DRL and ESC can be set adaptively to accommodate different channel dynamics.