Joint Sensing and Communications for Deep Reinforcement Learning-based Beam Management in 6G

The overall system model of this work is illustrated in Fig. 1. First, given the original dataset, we extract the image features of each episode in the dataset. Then, these features are used as input for a feed-forward neural network, which predicts the user locations. Next, considering the localization uncertainty, we propose a UK-medoids-based clustering method to form cluster groups for beam management. Finally, we apply DRL for the intra-beam resource allocation of each beam.

We consider a 5G-NodeB (gNB) serving a group of users with different QoS requirements. The UE set is denoted by U and each UE is represented by $U_{i}$. Based on observed locations, gNB will produce a certain number of UE clusters, and each cluster will be served by one beam. Here the inter-beam interference can be neglected due to the Orthogonal Frequency Division Multiple Access (OFDMA) technique. In each beam, radio resource is allocated to distribute the Resource Block Groups (RBGs) to the attached users.

As illustrated in Fig. 2, we assume two clusters are formed to cover 6 UEs. We assume the gNB receives distorted location information of the red car $U_{2}$, and two beams are formed accordingly. Therefore, due to the wrong beam direction, $U_{2}$ is covered by neither beam $b_{1}$ nor $b_{2}$, which results in packet loss and longer delay. This scenario explains how localization error can decrease the network performance of beamforming in mmWave networks. In this work, we mainly focus on the localization and clustering techniques, and a more detailed network and channel model can be found in [6].

The dataset used for localization is based on measurements from a communications system located at the University of Denmark [14]. In this dataset, one mobile vehicle drives over a region of 2.4 km by 1.25 km and provides measurements, including Signal to Interference and Noise Ratio (SINR), Reference Signal Received Power (RSRP), Reference Signal Received Quality (RSRQ), and Received signal strength indication (RSSI). Each sample contains one satellite image that shows the environment around the vehicle.

In digital imaging, the domain of the image is partitioned into $X$ rows and $Y$ columns during the sampling process. A pixel represents the area of intersection of one row and one column, which is considered as the smallest element in one picture, then the resolution of the image is $X\times Y$. In RGB color model, each pixel is defined by three values representing red, green, and blue ranging from 0 to 255.

The satellite images used in the localization process are RGB images with the same resolution of $256\times 256$, and each image is paired with a location sample. Therefore, we count the number of pixels that represent one specific object to compute the proportion of this object in the image. Since there are some common objects in the satellite images such as grass and buildings. We can use this pixel characteristic-based feature to characterize one image. In this work, we selected three objects. As shown in Fig. 3, the three selected objects are grass, gray building, and road. Then we counted the total number of pixels representing three objects respectively as three new features, which will be used to improve the localization accuracy.

To distinguish three objects, we adopted the method from [16]. We selected four color-based characteristics to observe their mean values, where the color characteristics are operations between three basic color values. As shown in Table I, the mean values of $|G-B|$ for the three categories are quite different. Moreover, the mean values of $|2G-R-B|$ are significantly different from the other two categories. After observing the histogram of these two values for three categories, we decide the value range for three categories as shown in Table II.

Finally, we deploy 7 features as input of feed forward neural networks (FFNN), including SINR, RSRP, RSRQ, RSSI, the number of pixels representing grass, gray building, and road. Then the FFNN will be trained to predict the user locations based on the input features [15].

UK-medoids is an advanced K-medoids based clustering technique [17]. It selects actual data points as cluster centers instead of computing the mean value of clusters. Meanwhile, the UK-medoids algorithm applies uncertain distance functions that are designed to better estimate the real distance between two uncertain objects.

It is presumed that there are $M$ positions to form $N$ clusters, and the $j^{th}$ cluster is denoted by $C_{j}$. The center of each cluster is denoted by $\boldsymbol{c_{j}}$. For each user position $\boldsymbol{p}$, the uncertainty is represented by a known probability distribution function (PDF) $f(\boldsymbol{p_{m}})$. We assume there are two uncertain objects $\boldsymbol{p_{m}}$ and $\boldsymbol{p_{n}}$, and the uncertain region is $R_{m}$ and $R_{n}$, respectively. Then the uncertain distance $\delta(\boldsymbol{p_{m}},\boldsymbol{p_{n}})$ is computed by:

where $dist(\boldsymbol{p_{m}},\boldsymbol{p_{n}})$ is a distance measure between $\boldsymbol{p_{m}}$ and $\boldsymbol{p_{n}}$. The uncertain distance is the expected value of the distance measurement between two uncertain objects.

Then, the medoid of a cluster is computed as the data point in the cluster with the least average dissimilarity to all other points in the cluster. The update method of a cluster center is:

In this paper, we assume the uncertainty region of all locations is specified as a uniform circle with center $\boldsymbol{p}$ and radius $R$. Then the PDF for a uniformly distributed circle is shown as below in the polar coordinate system:

where $r\in{[0,R]}$ and $\pi\in{[0,2\pi]}$.

In this way, equation (1) can be rewritten as:

which is the expansion of the previous equation (1) from the rectangular coordinate system to the polar coordinate system. In our work, we use Euclidean distance as our distance measurement.

The UK-medoids algorithm is summarized by Algorithm 1. First, all the uncertain distances between every two objects are computed. Note that these distances only need to be computed once during the whole algorithm, which will reduce the computation complexity. Then, the initial cluster centers are chosen, and it comes to the main loop. The main loop of UK-medoids contains two phases: the first phase is to allocate objects to a cluster, the second phase is to recompute cluster centers according to equation (2). The loop continues until there is no change in the cluster centers.

UK-Medoids will return a set of clusters, and we assume each cluster will be served by one beam[6]. Then DRL is performed for intra-beam radio resource allocation, which aims to offer high QoS requirements for both ultra-reliable low latency communications (URLLC) and enhanced mobile broadband (eMBB) users. The Markov decision process of DRL is defined as follows:

1) Actions: In each beam, the action is to allocate $i^{th}$ RBG to $j^{th}$ user.

2) States: The UE’s channel quality indicator (CQI) feedback is defined as states, which represents the channel condition between UEs and the BS.

3) Reward: URLLC and eMBB users have different QoS requirements, the reward function must take both requirements into account. URLLC users need minimal latency and high reliability, whereas eMBB users require a higher data rate. As such, the reward function has to consider the requirements of different types of users. The reward function is given by:

where $s_{i,b}$ is the SINR of the transmission link between $i^{th}$ RBG to a user in the $b^{th}$ beam, $s^{Tar}$ is the target SINR of eMBB users, $\tau^{Tar}$ is the target latency of URLLC users, and $\tau^{q}$ is the queuing delay. Here we use a target SINR to represent the high-reliability requirements of URLLC users. $sigm$ denotes the sigmoid function, which will normalize the reward:

Finally, we deploy Long Short-Term Memory (LSTM) network for Q-value prediction in DRL. As a special recurrent neural network, the LSTM network can better capture the long-term data dependency, which makes it an ideal application for complicated wireless environment.