Simultaneous Navigation and Radio Mapping
for Cellular-Connected UAV
with Deep Reinforcement Learning

Conventionally, cellular networks are designed to mainly serve terrestrial user equipments (UEs) with fixed infrastructure. With the continuous expansion of human activities towards the sky and the fast growing use of unmanned aerial vehicles (UAVs) for various applications, there have been increasing interests in integrating UAVs into cellular networks [2]. On one hand, dedicated UAVs could be dispatched as aerial base stations (BSs) or relays to assist wireless communications between devices without direct connectivity, or as flying access points (APs) for data collection and information dissemination, leading to a new three-dimensional (3D) network with UAV-assisted communications [3]. On the other hand, for those UAVs with their own missions, cellular network could be utilized to support their command and control (C&C) communication as well as payload data transmissions, corresponding to the other paradigm of cellular-connected UAV [4]. In particular, by reusing the existing densely deployed cellular BSs worldwide, together with the advanced cellular technologies, cellular-connected UAV has the great potential to achieve truly remote UAV operation with unlimited range, not to mention other advantages such as the ease of legitimate UAV monitoring and management, high-capacity payload transmission, and cellular-enhanced positioning [2, 4].

However, to practically realize the above vision of cellular-connected UAVs, there are still several critical challenges to be addressed. In particular, as existing cellular networks are mainly planned for ground coverage with BS antennas typically downtilted towards the ground [5], unlike that on the ground, ubiquitous cellular coverage in the sky cannot be guaranteed in general [6]. In fact, even for the forthcoming 5G or future 6G cellular networks that are planned to embrace the new type of aerial UEs, it is still unclear whether ubiquitous sky coverage is economically viable or not, even for some moderate range of altitude, considering various practical factors such as infrastructural and operational costs, as well as the anticipated aerial UE density in the near future. Such coverage issue is exacerbated by the more severe aerial-ground interference as compared to the terrestrial counterpart [6, 7, 4], due to the high likelihood of having strong line of sight (LoS) channels for the UAVs with their non-associated co-channel BSs.

Fortunately, different from the conventional ground UEs whose mobility is typically random and uncontrollable, which renders ubiquitous ground coverage essential, the mobility of UAVs is more predictable and most of the time controllable, either autonomously by computer program or by remote pilots. This thus offers a new degree of freedom to circumvent the aforementioned coverage issue, via coverage-aware navigation or trajectory design: an approach that requires no or little modifications for cellular networks to serve aerial UEs. There are some preliminary efforts along this direction. In [8], by applying graph theory and convex optimization for cellular-connected UAV, the UAV trajectory is optimized to minimize the UAV travelling time while ensuring that it is always connected with at least one BS. A similar problem is studied in [9, 10], by allowing certain tolerance for disconnection.

However, the conventional optimization-based UAV trajectory design like those mentioned above have practical limitations. First, formulating the corresponding optimization problems requires accurate and analytically tractable end-to-end communication models, including the BS antenna model, channel model, interference model and even local environmental model. Perhaps for such reasons, prior works such as [8, 9, 10] have mostly assumed some simplified models like isotropic radiation for antennas and/or free-space path loss channel model for aerial-ground links. Although there have been previous works considering more sophisticated channel models, like the probabilistic LoS channel model [11] and angle-dependent channel parameters [12, 13], these are statistical models that can only predict the performance in the average sense without providing any performance guarantee for the local environment where the UAVs are actually deployed. Another limitation of off-line optimization-based trajectory design is the requirement of perfect and usually global knowledge of the channel model parameters, which is difficult to acquire in practice. Last but not least, even with the accurate channel model and the perfect information of all relevant parameters, most of these off-line optimization problems are highly non-convex and thus difficult to be efficiently solved.

To overcome the above limitations, we propose in this paper a new approach for coverage-aware UAV navigation based on reinforcement learning (RL) [14], which is one type of machine learning techniques for solving sequential decision problems. As shown in Fig. 1, we consider a typical cellular-connected UAV system in urban environment, where multiple UAVs need to operate in a given airspace with their communications supported by cellular BSs. The UAV missions are to fly from their respective initial locations to a common final location with minimum time,¹¹1The results of this paper can be extended to the general case of different final locations for the UAVs. while maintaining a satisfactory communication connectivity with the cellular network with best effort. Our main contributions are summarized as follows:

• •

  First, we formulate the UAV trajectory optimization problem to minimize the weighted sum of its mission completion time and the expected communication outage duration, which, however, is difficult to be solved via the standard optimization techniques mainly due to the lack of an accurate and tractable communication model. Fortunately, we show that the formulated problem can be transformed to an equivalent Markov decision process (MDP), for which a handful of RL techniques, such as the classic Q learning [14], can be applied to learn the UAV action policy, or the flying direction in our problem.

• •

  As the resulting MDP involves continuous state space that essentially has infinite number of state-action pairs and thus renders the simple table-based Q learning inapplicable, we apply the celebrated deep neural network (DNN) to approximate the action-value function (or Q function), a technique commonly known as deep reinforcement learning (DRL) or more specifically, deep Q learning [15]. To train the DNN, we use the state-of-the-art dueling network architecture with double DQN (dueling DDQN) [16] and multi-step learning. Different from the conventional off-line optimization-based designs, the proposed DRL-based trajectory design does not require any prior knowledge about the channel model or propagation environment, but only utilizes the raw signal measurement at each UAV as the input to improve its radio environmental awareness, as evident from the continuously improved estimation accuracy of the UAVs’ action-value functions.

• •

  As direct RL based on real experience or measured data only is known to be data inefficient, i.e., a large amount of data or agent-environment interaction is needed, we further propose a new design framework termed simultaneous navigation and radio mapping (SNARM) to improve the learning efficiency. Specifically, with SNARM, the network (say, a macrocell BS in the area of interest) maintains a database, also known as radio map [17, 18], which provides the outage probability distribution over the space of interest with respect to the relatively stable (large-scale) channel parameters. As such, the obtained signal measurements as UAVs fly along their trajectories are used for dual purposes: to train the DQN to improve the estimation of the optimal action-value functions, as well as to improve the accuracy of the estimated radio map, which is used for the prediction of the outage probability for other locations in the considered area even if they have not been visited by any UAV yet. The exploitation of the radio map, whose accuracy is continuously improved as more signal measurements are accumulated, makes it possible to generate simulated UAV trajectories and predict their corresponding returns, without having to actually fly the UAVs along them. This thus leads to our proposed SNARM framework, by applying the classic Dyna [14] dueling DDQN with multi-step learning to train the action-value functions with a combined use of real trajectories and simulated trajectories, thus greatly reducing the actual UAV flights or measurement data required.

• •

  Numerical results are provided, with the complete source codes available online²²2https://github.com/xuxiaoli-seu/SNARM-UAV-Learning, to verify the performance of the proposed algorithms. It is observed that the SNARM algorithm is able to learn the radio map very efficiently, and both direct RL and SNARM algorithms are able to effectively navigate UAVs to avoid the weak coverage regions of the cellular network. Furthermore, thanks to the simultaneous learning of the radio map, SNARM is able to significantly reduce the number of required UAV flights (or learning episodes) than direct RL while achieving comparable performance.

Note that applying RL techniques for UAV communications has received significant attention recently [19, 20, 21, 22, 23] and DRL has also been used in wireless networks [24] for solving various problems like multiple access control [25], power allocation [26], modulation and coding [27], mobile edge computing [28], UAV BS placement [29], and UAV interference management [30], among others. On the other hand, local environmental-aware UAV path/trajectory design has been studied from different perspectives, such as that based on the known radio map [31] or 3D city map [32, 33], or the joint online and off-line optimization for UAV-enabled data harvesting [34]. Compared to such existing path/trajectory designs, the proposed coverage-aware navigation algorithms based on direct RL or SNARM do not require any prior knowledge or assumption about the environment, but only utilize the UAV’s measured data as the input for action and/or radio map learning, thus expected to be more robust to any imperfect knowledge or change of the local environment. Although large data measurements are generally required for the proposed algorithms, they are practically available according to the existing cellular communication protocols, such as the reference signal received power (RSRP) and reference signal received quality (RSRQ) measurements. Besides, as different UAVs in the same region share the same local radio propagation environment, regardless of operating concurrently or at different time, their measured data as they perform their respective missions can actually be collectively utilized for our proposed designs, to continuously improve the quality of the learning for the future missions, which thus greatly alleviates the burden for data acquisition in practice.

The rest of the paper is organized as follows. Section II introduces the system model and problem formulation. In Section III, we give a very brief background about RL and DRL, mainly to introduce the key concepts and main notations. Section IV presents the proposed algorithms, including the direct RL based on dueling DDQN with multi-step learning for UAV navigation, and SNARM to enable path learning with both real and simulated flight experience. Section V presents the numerical results, and finally we conclude the paper in Section VI.

As shown in Fig. 1, we consider a cellular-connected UAV system with arbitrary number of UAV users. Without loss of generality, we assume that the airspace of interest is a cubic volume, which is specified by $[x_{L},x_{U}]\times[y_{L},y_{U}]\times[z_{L},z_{U}]$, with the subscripts $L$ and $U$ respectively denoting the lower and upper bounds of the 3D coordinate of the airspace. UAVs with various missions need to operate in this airspace with communications supported by cellular BSs. While different UAVs usually have different missions and may operate at different time, since they share the common airspace, their flight experience including their signal measurements along their respective flying trajectories, can be exploited collectively to improve their radio environmental awareness. To make the problem concrete, we consider the scenario that the UAVs need to fly from their respective initial locations $\mathbf{q}_{I}\in\mathbb{R}^{3\times 1}$, which are generally different for different UAVs, to a common final location $\mathbf{q}_{F}\in\mathbb{R}^{3\times 1}$, as shown in Fig. 2. This setup corresponds to various UAV applications such as UAV-based parcel collection, with $\mathbf{q}_{I}$ corresponding to the customer’s home location and $\mathbf{q}_{F}$ being the local processing station of courier company, or UAV-based aerial inspection, with $\mathbf{q}_{I}$ being the target point for inspection and $\mathbf{q}_{F}$ being the charging station in the local area, etc. We assume that collision/obstacle avoidance is guaranteed by e.g., ensuring that UAVs are sufficiently separated in operation time or space, and their flight altitudes are higher than the maximum building height. Without loss of generality, we can focus on one typical UAV, since other UAVs may apply similar strategies to update the common location-dependent database, such as the action-value function of the RL algorithms and the radio map of the environment (to be defined later). The UAV needs to be navigated from its initial location $\mathbf{q}_{I}$ to the final location $\mathbf{q}_{F}$ with the minimum mission completion time, while maintaining a satisfactory communication connectivity with the cellular network. Let $T$ denote the mission completion time and $\mathbf{q}(t)\in\mathbb{R}^{3\times 1}$, $t\in[0,T]$, represent the UAV trajectory. Then we should have

where $\mathbf{q}_{L}\triangleq[x_{L};y_{L};z_{L}]$ and $\mathbf{q}_{U}\triangleq[x_{U};y_{U};z_{U}]$, with $\preceq$ denoting the element-wise inequality.

Let $M$ denote the total number of cells in the area of interest, and $h_{m}(t)$, $1\leq m\leq M$, represent the baseband equivalent channel from cell $m$ to the UAV at time $t$, which generally depends on the BS antenna gains, the large-scale path loss and shadowing that are critically dependent on the local environment, as well as the small-scale fading. The received signal power at time $t$ by the UAV from cell $m$ can be written as

where $\bar{P}_{m}$ denotes the transmit power of cell $m$ that is assumed to be a constant, $\beta_{m}(\cdot)$ and $G_{m}(\cdot)$ denote the large-scale channel gain and the BS antenna gain, respectively, which generally depend on the UAV location $\mathbf{q}(t)$, and $\tilde{h}_{m}(t)$ is a random variable accounting for the small-scale fading. As the proposed designs in this paper work for arbitrary channels, a detailed discussion of one particular BS-UAV channel model adopted for our simulations is deferred to Section V.

Denote by $b(t)\in\{1,\cdots,M\}$ the cell associated to the UAV at time $t$. We say that the UAV is in outage at time $t$ if its received signal-to-interference ratio (SIR) is below a certain threshold $\gamma_{\mathrm{th}}$, i.e., when $\mathrm{SIR}(t)<\gamma_{\mathrm{th}}$, where

Note that for simplicity, we have ignored the noise effect, since BS-UAV communication is known to suffer from more severe interference than the terrestrial counterparts, and thus its communication performance is usually interference-limited. Besides, we consider the worst-case scenario with universal frequency reuse so that all these non-associated BSs would contribute to the sum-interference term in (5). The extension of the proposed techniques for dynamic interference will be an interesting work for further investigation.

Due to the randomness of small-scale fading, for any given UAV location $\mathbf{q}(t)$ and cell association $b(t)$ at time $t$, $\mathrm{SIR}(t)$ in (5) is a random variable, and the resulting outage probability is a function of $\mathbf{q}(t)$ and $b(t)$, i.e.,

where $\Pr\left(\cdot\right)$ is the probability of the event taken with respect to the randomness of small-scale fading $\{\tilde{h}_{m}(t)\}_{m=1}^{M}$. Then the total expected outage duration across the UAV’s mission duration $T$ is

Intuitively, with longer mission completion time $T$, the UAV is more flexible to adapt its trajectory to avoid the weak coverage regions of the cellular network in the sky and thus reduce the expected outage duration $\bar{T}_{\mathrm{out}}$. Therefore, there in general exists a tradeoff between minimizing the mission completion time $T$ and expected outage duration $\bar{T}_{\mathrm{out}}$, which can be balanced by designing $\{\mathbf{q}(t)\}$ and $\{b(t)\}$ to minimize the weighted sum of these two metrics with certain weight $\mu\geq 0$, as formulated in the following optimization problem.

where $V_{\max}$ denotes the maximum UAV speed. It can be shown that at the optimal solution to $\mathrm{(P0)}$, the UAV should fly with the maximum speed $V_{\max}$, i.e., $\dot{\mathbf{q}}(t)=V_{\max}\vec{\mathbf{v}}(t)$, where $\vec{\mathbf{v}}(t)$ denotes the UAV flying direction with $\|\vec{\mathbf{v}}(t)\|=1$. Thus, $\mathrm{(P0)}$ can be equivalently written as

In practice, finding the optimal UAV trajectory and cell association by directly solving the optimization problem $\mathrm{(P1)}$ faces several challenges. First, to complete the problem formulation of $\mathrm{(P1)}$ to make it ready for optimization, we still need to derive the closed-form expression for the expected outage duration $\bar{T}_{\mathrm{out}}(\{\mathbf{q}(t),b(t)\})$ for any given UAV location $\mathbf{q}(t)$ and association $b(t)$, which is challenging, if not impossible. On one hand, this requires accurate and tractable modelling of the end-to-end channel $h_{m}(t)$ for cellular-UAV communication links, where practical 3D BS antenna pattern [35] and the frequent variations of LoS/NLoS connections, which depends on the actual building/terrain distributions in the area of interest, need to be considered. On the other hand, even with such modelling and the perfect knowledge of the terrain/building information for the area of interest, deriving the closed-form expression for $\bar{T}_{\mathrm{out}}(\{\mathbf{q}(t),b(t)\})$ is mathematically difficult since the probability density function (pdf) of the SIR depends on the UAV trajectory $\mathbf{q}(t)$ in sophisticated manners, as can be inferred from (4) and (5). Furthermore, even after completing the problem formulation of $\mathrm{(P1)}$ with properly defined cost function, the resulting optimization problem will be highly non-convex and difficult to be efficiently solved.

To overcome the above issues, in the following, we propose a novel approach for coverage-aware UAV navigation by leveraging the powerful mathematical framework of RL and DNN, which only requires the UAV signal measurements along its flight as the input. By leveraging the state-of-the-art dueling DDQN with multi-step learning [36], we first propose the direct RL-based algorithm for coverage-aware UAV navigation, and then introduce the more advanced SNARM algorithm to improve the learning efficiency and UAV performance.

This section aims to give a brief overview on RL and DRL, by introducing the key notations to be used in the sequel of this paper. The readers are referred to the classic textbook [14] for a more comprehensive description.

In this section, we present our proposed DRL-based algorithms for solving the coverage-aware UAV navigation problem (P1).

In this section, numerical results are provided to evaluate the performance of the proposed algorithms. As shown in Fig. 5, we consider an urban area of size 2 km $\times$ 2 km with high-rise buildings, which corresponds to the most challenging environment for coverage-aware UAV navigation, since the LoS/NLoS links and the received signal strength may alter frequently as the UAV flies (see Fig. 1). To accurately simulate the BS-UAV channels in the given environment, we first generate the building locations and heights based on one realization of the statistical model suggested by International Telecommunication Union (ITU) [40], which involves three parameters, namely, $\alpha_{\mathrm{bd}}$: the ratio of land area covered by buildings to the total land area; $\beta_{\mathrm{bd}}$: the mean number of buildings per unit area; and $\gamma_{\mathrm{bd}}$: a variable determining the building height distribution, which is modelled as Rayleigh distribution with mean value $\sigma_{\mathrm{bd}}>0$. Note that such statistical building model has been widely used to obtain the LoS probability for aerial-ground links [11], which, however, only reflects the average characteristics over a large number of geographic areas of similar type. While for each local area with given building locations and heights, the presence/absence of LoS links with cellular BSs can be exactly determined by checking whether the line connecting the BS and UAV is blocked or not by any building. Fig. 5 shows the 3D and 2D views of one particular realization of the building locations and heights with $\alpha_{\mathrm{bd}}=0.3$, $\beta_{\mathrm{bd}}=300$ buildings/km², and $\sigma_{\mathrm{bd}}=50$ m. For convenience, the building height is clipped to not exceed $90$ m.

We assume that the considered area has 7 cellular BS sites with locations marked by red stars in Fig. 5, and the BS antenna height is $25$ m [7]. With the standard sectorization technique, each BS site contains 3 sectors/cells. Thus, the total number of cells is $M=21$. The BS antenna model follows the 3GPP specification [5], where an 8-element uniform linear array (ULA) is placed vertically. Each array element itself is directional, with half-power beamwidths along the vertical and horizontal dimensions both equal to $65^{\circ}$. Furthermore, pre-determined phase shifts are applied to the vertical antenna array so that its main lobe is electrically downtilted towards the ground by $10^{\circ}$. This leads to the directional antenna array with fixed 3D radiation pattern, as shown in Fig. 4 of [2]. To simulate the signal strength received by the UAV from each cell, we firstly determine whether there exists an LoS link between the UAV and each BS at the given UAV location based on the building realization, and then generate the BS-UAV path loss using the 3GPP model for urban Macro (UMa) given in Table B-2 of [7]. The small-scale fading coefficient is then added assuming Rayleigh fading for the NLoS case and Rician fading with 15 dB Rician factor for the LoS case. It is worth mentioning that for a given environment, the aforementioned LoS condition, antenna gain, path loss and small-scale fading with each BS are all dependent on the UAV location in a rather sophisticated manner. This makes them only suitable to generate the numerical values with given UAV location, instead of being directly used for UAV path planning with the conventional optimization technique as for solving $\mathrm{(P1)}$, even with accurate channel modelling and perfect global information. This thus justifies the use of RL techniques for UAV path design as pursued in this paper.

For both Algorithms 1 and  2, the dueling DQN adopted is a fully connected feedforward ANN consisting of 5 hidden layers. The numbers of neurons of the first 4 hidden layers are 512, 256, 128, and 128, respectively. The last hidden layer is the dueling layer with $K+1$ neurons, with one neuron corresponding to the estimation of the state-value and the other $K$ neurons corresponding to the action advantages for the $K$ actions, i.e., the difference between the action values and the state value of each state. The outputs of these $K+1$ neurons are then aggregated in the output layer to obtain the estimation of the $K$ action values [16], as shown in Fig. 3. For SNARM in Algorithm 2, the ANN for radio mapping also has 5 hidden layers, with the corresponding numbers of neurons given by 512, 256, 128, 64, and 32, respectively. The activation for all hidden layers is by ReLu [37], and the ANN is trained with Adam optimizer to minimize the mean square error (MSE) loss, using Python with TensorFlow and Keras.³³3https://keras.io/ For ease of illustration, we assume that the UAV’s flying altitude is fixed as $H=100$ m, and the number of actions (or flying directions) at each time step is $K=4$. All UAV missions have a common destination location $\mathbf{q}_{F}=[1400,1600,100]^{T}$ m, while their initial locations $\mathbf{q}_{I}$ are randomly generated. The transmit power of each cell is assumed to be $\bar{P}_{m}=20$ dBm and the outage SIR threshold is $\gamma_{\mathrm{th}}=0$ dB. Other major simulation parameters are summarized in Table I.

Fig. 6(a) shows the true global coverage map for the considered area, i.e., the coverage (non-outage) probability for each location by the cellular network, where coverage probability is the complementary probability of the outage probability defined in (6). Note that such a global coverage map is numerically obtained based on the aforementioned 3D building and channel realizations via computer simulations, while it is not available for our proposed Algorithm 1 or Algorithm 2. It is observed from Fig. 6(a) that the coverage pattern in the sky is rather irregular due to the joint effects of the 3D BS antenna radiation pattern and building blockage. It is observed that around the center of the area, there are several weak coverage regions with coverage probability less than $30\%$. Obviously, effective coverage-aware UAV navigation should direct UAVs to avoid entering such weak coverage regions with the best effort.

To validate the quality of radio map estimated by the SNARM framework proposed in Algorithm 2, Fig. 6(b) shows the finally learned coverage map by Algorithm 2. By comparing Fig. 6(a) and Fig. 6(b), it is observed that the learned coverage map is almost identical to the true map, with only slight differences. This thus validates the effectiveness of applying SNARM for the radio map estimation and coverage-aware path learning based on it. This is further corroborated by Fig.7, which shows the MSE and mean absolute error (MAE) of the learned radio map versus the episode number. Note that the MSE and MAE are calculated by comparing the predicted outage probabilities using the learned radio map versus their actual values in the true map for a set of randomly selected locations. It is observed that the quality of the learned radio map is rather poor initially, but it improves rather quickly with the episode number as more signal measurements are accumulated. For example, with only 200 episodes, we can already achieve 87.1% of the MSE reduction that is achieved after 5000 episodes.

As for the performance of the proposed learning algorithms, Fig. 8 shows the moving average return per episode for the direct RL in Algorithm 1 and SNARM in Algorithm 2, where the moving window has the length of 200 episodes. For the SNARM algorithm, the number of steps $\tilde{N}$ based on simulated experience per actual step at episode $n$ is set as $\tilde{N}=\min(\lfloor n/100\rfloor,10)$, so that $\tilde{N}$ increases with the episode number and converges to 10. Recall that the return of each episode for RL is the accumulated rewards for all time steps in the episode. It is observed that though experiencing certain fluctuation, as usually the case for RL algorithms, both proposed algorithms demonstrate an overall tendency of increasing average return. Furthermore, recall that the dueling DQNs are initialized to encourage shortest path flying for the early episodes when the UAV has completely no knowledge about the radio environment, which leads to an average return about $-2700$ initially. After training for 5000 episodes, the resulting return has increased to $-1089$ and $-1486$ for SNARM and direct RL, respectively. Another observation from the figure is that the direct RL has faster improvement over the SNARM algorithm at the earlier episodes. This is probably due to the insufficient accuracy of the learned radio map for SNARM, which may generate inefficient simulated experience. However, as the quality of the radio map improves, the SNARM algorithm achieves better performance than direct RL, thanks to its exploitation of the learned radio map for path planning. In practice, such a performance improvement translates to fewer agent-environment interactions, which is highly desirable due to the usually high cost of data acquisition from real experience. For example, to achieve an average return of $-1500$, Fig. 8 shows that about 3200 actual UAV flight episodes are required for direct RL in Algorithm 1, but this number is significantly reduced to about 1000 with SNARM in Algorithm 2.

Last, Fig. 9 shows the resulting UAV paths with the proposed algorithms for the last 200 episodes of learning, which correspond to 200 randomly generated initial locations, as denoted by red crosses in the figure. The common destination location $\mathbf{q}_{F}$ is labelled by a big blue triangle. Also shown in the figure is the true global coverage map. It is observed that both direct RL and SNARM algorithms are able to direct the UAVs to avoid the weak cellular coverage regions with the best effort, as evident from the generally detoured UAV paths as well as the much sparser paths in areas with low coverage probabilities. For example, as can be seen from Fig. 9(b), the SNARM algorithm is able to discover and follow the narrow “radio bridge” located at the x axis around 1000 m and the y axis from about 1000 m to 1700 m. This shows the peculiar effectiveness of SNARM for coverage-aware UAV path learning.

This paper studies coverage-aware navigation for cellular-connected UAVs. To overcome the practical limitations of the conventional optimization-based path design approaches, we propose DRL-based algorithms, which only require UAV signal measurements as the input. By utilizing the state-of-the-art dueling DDQN with multi-step learning, a direct RL-based algorithm is first proposed, followed by the more advanced SNARM framework to enable radio mapping and reduce the real UAV flights for data acquisition. Numerical results are provided to show the effectiveness of the proposed algorithms for coverage-aware UAV navigation, and the superior performance of SNARM over direct RL based navigation.