Toward Integrating Semantic-aware Path Planning and Reliable Localization for UAV Operations

To ensure the ability of the real-time performance of our proposed system, we first introduce a keyframe decision architecture as shown in Fig. 1 to evaluate the usefulness of RGB images. Additionally, semantic segmentation is a challenging task that may require considerable time to execute and infer results, leading to non-continuous UAV flight while the segmentation of all frames is unnecessary. Therefore, our key frame decision module determines whether the image contains sufficient or rich semantic information to be fed into the semantic segmentation module, which requires much less execution time than the semantic segmentation module.

To achieve this goal, we segment the image into distinct regions to capture comprehensive information from various areas. This segmentation is akin to the patching process utilized in contemporary vision transformer models, where the input image I of size $H\times W$ is divided into non-overlapping smaller patches of size $P\times P$. Mathematically, this segmentation can be expressed as

where $\textbf{I}_{i,j}$ denotes the patch located at the $i-th$ row and $j-th$ column of the patch grid.

Subsequently, data from each patch is globally average pooled to condense the information, with the global average pooling operation for a given patch $\textbf{I}_{i,j}$ computed as:

The pooled image is then flattened into a one-dimensional vector for further processing, described as $F(I)=concat\big{(}GAP(\textbf{I}_{1,1},GAP(\textbf{I}_{1,2},\cdots,GAP(\textbf{I}_{\frac{H}{P},\frac{W}{P}})\big{)}$. This vector is passed through a fully connected layer for evaluation, modeled by $\textbf{y}=\sigma(\textbf{W}\textbf{x}+b)$, where W is the weight matrix, $b$ is the bias vector, and $\sigma$ denotes the activation function. The architecture ensures meticulous assessment of pixel value distribution across image regions. By employing only a patching layer and a fully connected network, the system achieves minimal execution time, enabling real-time operation. This module is crucial for meeting the system’s requirements, ensuring efficient and accurate UAV localization by leveraging the robustness of semantic segmentation and enhancing computational efficiency, making it suitable for deployment in dynamic and complex environments.

After selecting the keyframe, semantic segmentation plays a vital role in extracting meaningful areas of information from the surrounding environment. Prior research, including PSPNet [19] and DeepLab [20] has demonstrated effective semantic segmentation using pre-trained backbone networks such as VGG16 [21], and ResNet [22] variants (e.g. ResNet50, ResNet100), and MobileNet [23]. These architectures enhance model accuracy by expanding the receptive field through techniques like the Pyramid Pooling Module and Atrous Convolutions, which are employed during the encoding phase to capture broad semantic context. In the decoding phase, conditional random fields are used to smooth the output, with the aim of improving the accuracy and execution time. However, the execution time of both the DeepLab and PSPNet models remains relatively high due to their sequential nature, traversing each layer sequentially in the encoding and decoding phases. On the other hand, edge information is crucial for semantic segmentation, as it delineates boundaries between semantic regions within an image. Accurate preservation of edge information, typically found in the initial layers of backbone networks such as ResNet, InceptionNet, and VGG16, is essential for optimal model performance. Models like Unet and PsPNet prioritize edge information by using skip connections to transfer features from the encoding to the decoding module, enhancing the final segmentation accuracy.

To address these challenges, we propose a model (Semantic Segmentation module in Fig. 1) that supports parallel computation while leveraging edge information and maintaining rapid processing capabilities suitable for real-time systems. Our approach integrates the strengths of DeepLabv3, particularly its use of atrous convolutions, which accelerate processing and minimize the loss of critical features during pooling. Our model extracts edge information and incorporates it into the feature map at the final working layer. Additionally, features from other intermediate layers are combined with the feature map, thus requiring only two decoding layers compared to the more extensive structures in SegNet and DeepLabv3. This approach significantly improves execution time while maintaining high accuracy.

Moreover, the attention mechanism shown in Fig. 2 enhances the segmentation performance by focusing on the relevant parts of the images, capturing fine details and context. The attention architecture involves several steps. First, global pooling aggregates contextual information across the entire feature map:

where g is the global context vector, $H$ and $W$ are the height and width of the feature map, and $\textbf{f}_{ij}$ represents the feature vector at position $(i,j)$.

Next, the context vector g is passed through $1\times 1$ convolution $\textbf{g'}=\textbf{W}_{1}\textbf{g}+b_{1}$ with $\textbf{W}_{1}$ and $b_{1}$ is the weight matrix and bias of the $1\times 1$ convolution. The original feature map f is processed through a $3\times 3$ convolution to emphasize important features: $\textbf{f'}=\textbf{W}_{2}*\textbf{f}+b_{2}$ with $\textbf{W}_{2}$ and $b_{2}$ the weight matrix and bias of the $3\times 3$ convolution, respectively. The attention weights w are computed as $\textbf{w}=\textbf{f'}\odot\textbf{g'}$ with $\odot$ denoting element-wise multiplication. Finally, the refined feature map $\textbf{f}_{out}$ is obtained by adding these attention weights back to the original feature map: $\textbf{f}_{out}=\textbf{f}_{i,j}+\textbf{w}_{i,j}$.

To further boost performance, we employ an auxiliary loss technique, where the combined loss function is:

where $\lambda$ balances the primary loss, $\mathcal{L}$ represents the discrepancy between the predicted output and the reference, and $\mathcal{L}_{\text{aux}}$ is the auxiliary loss function computed from the top layer of the decoder model.

Upon receiving the semantic map, we generate a cost map based on the normalized semantic information. Since we have 23 labels, the cost map value ranges from 0 to 1 corresponding to the semantic mask value from 0 to 22. We then present a hierarchical planner that incorporates the Dynamic Window Approach (DWA) [24] algorithm with this cost map. This algorithm aims to determine the optimal pair of linear and angular velocity values that describe the best achievable trajectory for the robot within the next time step. This is achieved through two main steps:

1. 1.

   Determining feasible velocities based on constraints related to acceleration, deceleration, and obstacle avoidance.

2. 2.

   Selecting the velocities that optimize an objective function.

The objective function of the Semantic-aware DWA controller, considering the cost map, is defined as follows:

where $\alpha,\beta,\gamma,\epsilon$ are positive weight coefficients, and $H(v,\omega),D(v,\omega),Vel(v,\omega)$ represent heading function, distance function and velocity function, respectively. The cost map function $C(v,\omega)$ is defined as:

where $t$ is the flight time and $f_{k}(x,y)$ is the value at the $k-th$ discrete position $(x,y)$ of the robot on the trajectory mapped onto the cost map.

Thus, the Semantic-aware DWA path planning system accounts for dynamic constraints, motion capabilities, and environmental semantic factors. The parameter $e^{-0.2t}$ indicates that initially, the UAV can choose a longer flight path to reach a better semantic area. However, as it approaches the destination, it needs to fly directly to the destination as quickly as possible, as positioning errors become less critical due to their cumulative nature.

Our proposed semantic segmentation model is trained on the UAV Image Dataset, which comprises 3,600 images captured from UAVs at altitudes ranging from low to medium (20-30m). This dataset includes 23 semantic labels that encompass crucial labels utilized in this study, such as water bodies, grasslands, cars, and high-rise buildings.

We evaluated the efficacy of the keyframe decision module using accuracy metrics, considering the binary classification nature of the problem. The training dataset consisted of 200 images, with 137 images labeled as “Yes” and 63 images labeled as “No”, as shown in Fig. 4. The key frame decision module achieved an accuracy of 72% and a true positive for the “Yes” label reached $0.83$, demonstrating its capability to retain important cases that necessitate passing through the semantic segmentation module.

Table 1 compares our semantic segmentation model with previous models, using metrics such as mIoU (mean Intersection over Union) and average execution time (AET), averaged over 20 measurements. Our proposed model outperforms standard models like PSPNet [19], FPN [14], and FAN [13]. Specifically, it shows a $22.6\%$ improvement in mIoU compared to PSPNet, $21.3\%$ compared to FPN, and $15.4\%$ compared to FAN. Additionally, it enhances execution time compared to these models. While maintaining similar execution times to DeepLabv3 [20], our model demonstrates an $8\%$ accuracy improvement. Compared to the Transformer model Segformer [15], our model ensures a satisfactory execution time with only a slight decrease in accuracy.

To demonstrate the ability of navigation, we calculated the UAV’s flight distance and the distance flown into areas with unreliable localization. Specifically, unreliable localized areas in the Singapore environment include roads, water bodies, and the vicinity of high-rise buildings, while in the Baxall environment, poorly localized areas are defined as roads and cars. The results are summarized in Table 2:

The flight distance of the Semantic-aware DWA system slightly climbed over the standard DWA in both test environments, increasing by $11.6\%$ in the Baxall environment and $42.5\%$ in the Singapore environment. Furthermore, the unreliable distance of the UAV’s flight path shows substantial improvement, with a $52.6\%$ and $99.0\%$ increase in the Baxall and Singapore environments, respectively. This highlights the system’s precision and reliability. The integration of DWA with semantic segmentation also reduces positioning errors during the initial periods, proving beneficial for UAV systems by mitigating the accumulation of errors along the flight path.