Task-Oriented Image Transmission for Scene Classification in Unmanned Aerial Systems

In the era of Internet of Everything, communication plays an important role in the collaborative computing tasks of multiple sensors and computing resources. It is necessary to study the application of task-oriented communication in the future IoT paradigm, which dynamically adjusts the information transmission strategy adapted to the task objective and the underlying changes of the communication environments. The works in [6, 7] started from Shannon and Weaver’s original definition and categorization of communication, discussing the value of information while proposing the vision of task-oriented communication and its effectiveness from the perspectives of information, communication, control theory, and computer science.

With the rapid improvement of hardware performance, more and more deep learning models are applied in the communication scenarios, even in the task-oriented communication systems [15, 16, 17, 18, 19, 20]. Xie et al. proposed the deep semantic communication (DeepSC) method for wireless text transmission [16], and its variations, L-DeepSC [15], DeepSC-S [17], and MU-DeepSC [19] for complexity reduction, speech transmission, and multi-user transmission with multi-modal data. Particularly, L-DeepSC [15] extends DeepSC to a lite mode, which is affordable for power-constrained IoT devices. [17] is designed for speech signal transmission by realizing semantic information exchange through an attention mechanism. Moreover, [20] extends DeepSC from point to point transmission to a multi-user transmission system, in which the source encoder and channel encoder are both composed of deep networks. Particularly, L-DeepSC [15] extends DeepSC to a lite mode, which is affordable for power-constrained IoT devices. [19] is designed for speech signal transmission by realizing semantic information exchange through an attention mechanism. Moreover, [20] extends DeepSC from point to point transmission to a multi-user transmission system for supporting multimodal data transmission. [17] subsequently introduced Reed Solomon (RS) channel coding to enhance the reliability of transmission. Similarly, an adaptive circular transformer is applied to semantic communication for more flexible transmission [18].

Reinforcement learning (RL) is a classic machine learning method which constructs a Markov decision process (MDP) where agents acquire rewards from environment to update the model parameters in a trial-and-error manner. Deep reinforcement learning (DRL) combines the perception ability of deep learning with the decision-making ability of reinforcement learning. In the high-dimensional continuous space of the state-action pair space, value-based learning will encounter difficulties, and the strategy-based reinforcement learning shows better convergence [21, 22].

A growing number of researchers are applying DRL to solve the perception-decision problem in computer vision tasks [23]. [24] applies a policy gradient based on random curiosity-driven exploration to image coding, resulting in good performance in Atari games. Uzkent et al. select HR image blocks with policy gradients to improve classification accuracy with LR images as states [25]. Yuan et al. control the compression of the target model with agent exploring the relationship between the input and output [26]. Hu et al. jointly optimize the offloading decision and video coding parameters of the MEC-based face recognition framework with Q-learning [27]. Li et al. improve the bit rate by selecting quantization parameters (QPs) for different coding units (CUs) with policy gradients [28]. Li et al. propose an image cropping model based on weakly-supervised DRL in [29]. In the same way, Zhang et al. construct two agents to move and zoom the bounding-box with the emotional and background information in the image [30]. By simulating an agent as a brush, Zhang et al. sequentially relight portraits by DRL and learned a coarse-to-fine policy with local interpretability [31]. Yarats et al directly construct a value function in the pixel domain and regularize the learned policy with model-free DRL, so that the image has better features and visual effects [32].

Semantic blocks are defined as a collection of image blocks that contribute the most to the correct classification of the back-end model. The semantic extraction module is to extract effective semantic blocks for compression through the perception of image content and channel gain, which mainly includes a policy network. Due to the limited computing resources on the sensor, directly using HR images for semantic extraction will cause high energy consumption and latency. Aided by the principle of CS sampling, the front-end sensor can capture LR images in advance, which can be regarded as a prerequisite for semantic extraction. As a branch of joint perception, LR images can also provide semantic information about HR images to guide the model to find out critical pixel regions. The other channel sensing branches are able to provide information for channel gains, enabling the model to select semantic blocks at different scales. Then, the output policy network for semantic extraction is defined as follows:

where $\textbf{X}^{lr},g$ denotes the input LR image and channel gain respectively. $f_{\theta}$ is a ResNet18 for image feature extraction while $f_{\phi}$ is a multilayer perceptron (MLP). $f_{\psi}$ is also a MLP for joint decision-making. $\theta,\phi,\psi$ respectively indicates the corresponding parameters of these parts.

As shown in Fig. 2, it is assumed that the HR images are divided into 4 $\times$ 4 blocks, where each block provides different semantics for the back-end model. The semantic extraction module can choose any combination of them. Therefore, p in Eq. (1) defines sixteen Bernoulli distributions where $\textbf{p}[i]$ indicates the probability of each block in the image being a semantic at each dimension $i$. The actions performed by the policy network can be sampled at each dimension $\textbf{a}[i]\sim\left\{0,1\right\}^{\textbf{p}[i]}$, where $\textbf{a}[i]=1$ indicates that the $i$-th semantic block is extracted for compression and $\textbf{a}[i]=0$ otherwise. Therefore, the joint probability density function of the output of the policy network can be expressed as

Then we would model the policy network as a single-step MDP, and the detailed optimization process will be introduced in Sec. IV.

Compressed sensing, also called sparse sampling, is a technique for finding sparse solutions of underdetermined linear systems, which has been widely used in the field of signal processing in the past two decades. It maps a linear signal to the compressed domain through a sampling matrix. For high-dimensional image signals, the sampling procedure is performed block by block:

where $\textbf{x}_{i}\in{{\mathbb{R}}^{n}}$ is the vector form of $i$’th sub-block in pixel domain. $\textbf{y}_{i}\in{{\mathbb{R}}^{m}}$ is the compressed signal of $\textbf{x}_{i}$. $\boldsymbol{\Phi}\in{{\mathbb{R}}^{m\times{n}}}(m\ll{n})$ denotes a sampling matrix, which is implemented by a learnable convolution kernel in our system. Note that the sub-block in CS is different from the semantic blocks introduced in previous section.

Traditional methods of recovering the signal $\textbf{x}_{i}$ need to solve a convex optimization problem of $l_{1}$-norm, which is usually of high complexity [33]. In recent years, with the rise of deep learning, some researchers have used CNN to reconstruct the signal $\textbf{x}_{i}$ [34]. Since the sampling matrix can also be implemented through a convolutional layer, both compression and reconstruction steps can be trained through an end-to-end CNN. In our system, $\textbf{x}_{i}\in{\mathbb{R}}^{k\cdot{k}\cdot 3}$ is a vector reshaped from an image block $\textbf{X}_{i}\in{\mathbb{R}}^{k\times{k}\times 3}$ with channels RGB, and $k$ denotes the height and width of the sampled sub-blocks. We use a kernel to convolve the image:

where $\boldsymbol{\kappa}\in{\mathbb{R}}^{k\times{k}\times 3\times{m}}$ is the convolution kernel without bias, $m=\lceil{k}\cdot{k}\cdot 3\cdot{sr}\rceil$, $sr$ is the sampling rate. $\textbf{X}=stitch\{\textbf{X}_{1},\textbf{X}_{2},\cdots,\textbf{X}_{P}\},\textbf{Y}=stitch\{\textbf{Y}_{1},\textbf{Y}_{2},\cdots,\textbf{Y}_{P}\}$. $\star$ indicates a convolution operation with a stride of k. $stitch$ means to stitch all image blocks into a whole image. $P$ is the total number of image blocks. The elements in the convolution kernel $\boldsymbol{\kappa}$ and the sampling matrix $\boldsymbol{\Phi}$ are one-to-one correspondence. The two procedures described by Eq. (3) and Eq. (4) are equivalent.

A denoising-based approximate message passing algorithm (D-AMP) was proposed in [33], which alternately iterates the residual variables and the reconstructed signal through a denoising function:

where $\textbf{z}^{l}$ is the iterative residual in the compressed domain, $D_{\hat{\sigma}}$ is a Gaussian denoising filter with parameter $\hat{\sigma}$. ${\rm{div}}D_{\hat{\sigma}^{l-1}}{(\cdot)}$ is the divergence of the output Gaussian distribution for the correction of the residual. ${\boldsymbol{\Phi}}^{-1}_{\rm{left}}$ is the left inverse matrix of $\boldsymbol{\Phi}$.

In order to use CNN to replace the residual correction term to avoid complicated divergence calculations, we can use a residual structure CNN to predict a new residual from the original term, as illustrated in Fig. 3. Suppose that the residual consists of $L$ iterations, in each iteration $\hat{\textbf{X}}^{l-1}$ represents the reconstructed image output from the previous stage. The residuals for each sub-block in the compressed domain $\{\textbf{y}_{1}-\boldsymbol{\Phi}\hat{\textbf{x}}_{1}^{l-1},\textbf{y}_{2}-\boldsymbol{\Phi}\hat{\textbf{x}}_{2}^{l-1},\cdots,\textbf{y}_{N}-\boldsymbol{\Phi}\hat{\textbf{x}}_{N}^{l-1}\}$ can be calculated through the sampling matrix. The residual iteration in the compressed domain may still cause serious blocking effects. Mapping the residuals back to the pixel domain with ${\boldsymbol{\Phi}}^{-1}_{\rm{left}}$ for correction can solve this problem:

where $\textbf{e}_{i}^{l-1}$ denotes the residual vector of $i$’th image block at $(l-1)$’th stage, $\textbf{E}^{l-1}=stitch\{\textbf{E}_{1}^{l-1},\textbf{E}_{2}^{l-1},\cdots,\textbf{E}_{P}^{l-1}\}$ is the residual image at $(l-1)$’th stage. $\boldsymbol{\kappa}^{-1}\in{\mathbb{R}}^{k\times{k}\times{m}\times 3}$ represents the kernel corresponding to $\boldsymbol{\Phi}^{-1}_{\rm{left}}$, $\overline{\star}$ indicates the transposed convolution with a stride of $k$.

Then update the residual in the pixel domain

where $\boldsymbol{\omega}^{l}$ is the parameters of the residual network with two convolutional layers. $\textbf{Z}^{l}$ represents the updated residuals at $l$-th stage.

Finally, the image $\hat{\textbf{X}}^{l}$ recovered at this stage is output by the denoising network:

where $\boldsymbol{\Omega}^{l}$ indicates the parameters of the denoising network with three convolutional layers.

Since only the semantic part of the image is transmitted, but the reconstruction procedure is performed on the entire image, the compressed values of the non-semantic part is set to 0 to reduce the computational complexity of the reconstruction network. Therefore, the parameters of all layers do not contain bias.

The transmission latency of the system is mainly determined by the transmission rate and the number of bits transmitted. The transmission latency of the uplink is considered in the proposed system. We assume that the channel between the drones and the BS follows the stationary Rayleigh fading distribution, where the channel gain and transmission rate remain static within a small time interval. Then, the uplink transmission rate at time slot $t$ is as follows:

where $w_{D\rightarrow{B}}$ denotes the BS bandwidth allocated to drones. The signal-to-interference-plus-noise ratio (SINR) for the signals received at the BS is expressed as $\rm{SINR}={p(t)g(t)}/{N_{0}}$, where $p(t)$ is the transmission power of drones. The additive white Gaussian noise (AWGN) is $N_{0}$. The channel gain for the signals received at the BS is $g(t)=d^{-\alpha}u_{t}$ for instantaneous perception of communication environment. The distance $d$ between drones and BS is the most important factor affecting the channel gain. We will consider the path loss exponent $\alpha$ and Rayleigh distribution $u_{t}$ with unit mean. Generally, channel gain is an unlimited continuous variable. It can be quantified in $g(t)\in${-30 dB, -20 dB, -10 dB, 0 dB, 10 dB, 20 dB, 30 dB}, making it easier for the model to discriminate.

Assuming that the policy network has extracted a total of $N^{a}$ semantic blocks, the number of bits required to be transmitted is given by

where $H$ and $W$ represent the height and width of the HR images. $\gamma$ implies number of quantization bits in compressed domain.

Then the uplink transmission latency is given by

As mentioned above that the transmission rate $r_{D\rightarrow{B}}(t)$ in each time slot under a stationary channel is constant, the latency is approximately the ratio of the number of transmitted bits $N^{b}$ to $r_{D\rightarrow{B}}(t)$.

As described in Sec. III, the policy network is modeled as a single-step MDP. At any time, the image captured by the sensor and the channel gain sensed in real time are two independent variables. For the proposed model, the image is a high-dimensional continuous variable while the channel gain is a single discrete value. The output policy p implies $2^{16}$ discrete actions that may be performed by the agent. It is tricky to optimize our high-dimensional system with traditional reinforcement learning methods based on value functions. Policy gradient is not only suitable for our high-dimensional system, but also capable of leanring random policies that cannot be learned by the value function. The purpose is to optimize the parameters $\theta,\phi,\psi$ by maximizing the expected value of cumulative rewards on all policy trajectories:

where $\tau\sim\pi(\tau)$ is a trajectory of states, actions, and rewards sampled from an interaction with our policy, and $r(\tau)$ represents the sum of all rewards for this trajectory.

For our single-step MDP process, trajectory $\tau$ is composed of $\textbf{X}^{lr},g$ and a. The objective is to maximize the accuracy of the back-end target model while minimizing the uplink transmission latency of the system by maximizing a reward function:

where $M$ and $N$ respectively represent the total number of $\textbf{X}^{lr}_{m}$ and $g_{n}$. $R_{m,n}$ indicates the single-step reward from the system with the input of $\textbf{X}^{lr}_{m}$ and $g_{n}$, which is defined by the system’s transmission latency $T_{m,n}$ and the correctness of the target model

where $f_{k}^{tar}$ denotes the $k$-th output of the backend target model. $C$ is the label of input X, and $\mathcal{R}(T_{m,n})$ is a function negatively correlated to $T_{m,n}$. A penalty term is represented by $\eta$. Two kinds of $\mathcal{R}(T_{m,n})$ are tried in proposed system

where $\lambda$ is a positive hyperparameter.

The gradients of $\theta,\phi,\psi$ can be updated along the gradient ascent. To avoid over-update of the network with great rewards and keep training stable, an offset need to be subtracted:

where $\overline{R}_{m,n}$ is the system reward when performing a baseline action $\overline{\textbf{a}}$, which directly selects the image blocks according to the larger value of the output policy without sampling

Since the parameters of the policy network are divided into three parts, and the two input variables are independent of each other, we use the control variable method to train it. This optimization process is divided into three stages. First, the channel gain is kept changing and $\phi$ are kept constant in the outer loop, while $\theta$ and $\psi$ are updated with different LR images in the inner loop. In the middle stage, the LR images are kept changing and $\theta$ are kept constant in the outer loop, while $\phi$ and $\psi$ are updated with different channel gains in the inner loop. In the last stage, $\theta,\phi,\psi$ are updated synchronously with the LR images and channel gains being sampled randomly.

As illustrated in Sec. III, the sampling and compression processes of CS are carried out simultaneously. By deploying the sampling matrix $\boldsymbol{\Phi}$ in the front-end UAV, the measured values of semantic blocks in compressed domain can be obtained quickly. The measurement matrices of traditional CS algorithms are usually Gaussian matrices or Hadamard matrices, which should be constructed to satisfy Restricted Isometry Property (RIP) conditions. In proposed system, $\boldsymbol{\Phi}$ can be implemented through a convolutional layer and an optimal parameter within the kernel can be learned over the data distribution. Because the deep reconstruction procedure requires the auxiliary calculation of the inverse matrix $\boldsymbol{\Phi}^{-1}_{\rm{left}}$, it is necessary to pretrain the sampling matrix $\boldsymbol{\Phi}$ through an auxiliary layer due to the underivativity of the inversion operation. Assuming that the transposed convolution kernel of the auxiliary layer is $\kappa^{\prime}\in{\mathbb{R}}^{k\times{k}\times{m}\times 3}$ with a stride of $k$, an initial reconstruction process can be given by

where Eq. (4) and Eq. (20) compose a pair of compressing and preliminary reconstruction process. The pre-reconstructed image can be approximated by minimizing the mean square error (MSE) loss between X and $\hat{\textbf{X}}^{0}$:

Once the optimal sampling kernel $\boldsymbol{\kappa}$ is learned, $\boldsymbol{\kappa}^{-1}$ can be calculated according to the inverse matrix $\boldsymbol{\Phi}_{\rm{left}}^{-1}$. The pre-reconstructed image $\hat{\textbf{X}}^{0}$ for each HR image can be calculated through this auxiliary, which can be used to optimize the deep reconstruction layers in the subsequent stages. Given the set of original images and reconstructed images $\{\textbf{X}_{m},\hat{\textbf{X}}_{m}^{L}\}_{m=1}^{M}$, the parameters in multi-stage reconstruction layers can also be updated with MSE loss:

Although only semantic blocks are compressed and transmitted, in order to avoid the block effect caused by reconstruction, the training process of the entire reconstruction network is carried out on the full images, without any special operations for any macroblock. Since the reconstructed network does not contain bias, setting the measurement values of the non-semantic regions to 0 during reconstruction will not affect the reconstruction quality too much.

When optimizing the entire system, the two modules are updated in an alternate iterative manner. First, the deep CS model is trained with the full HR images. Afterwards, the semantic extraction model can be trained by this model, since the calculation of system rewards depends on the quality of compression reconstruction. Driven by policy network trained at the previous stage, the deep CS model can be finetuned with HR images.

In this section, the experimental setup for the proposed scheme has been introduced. The experiments are conducted in an Ubuntu operating system (CPU Intel core i7-10700 2.9 GHz; memory 64GB, GPU NVIDIA GeForce RTX 3090, which contains 10496 CUDA computing core units and 24GB graphics memory). We use Python and deep learning framework of Pytorch for simulation.

The transmission power of the drones, the noise of the channel and the bandwidth are fixed to 15 dBm, -97 dB, and 100 KHz, respectively. We divide the channel gains into 7 states as the input of the perception network: $g(t)\in\{-30,-20,-10,1,10,20,30\}$ dB. As a result, their corresponding 7 uplink transmission rates are $r_{D\rightarrow{B}}(t)\in\{105,355,675,1006,1338,1670,2003\}$ kbps.

We evaluated the performance of the proposed algorithm on the Aerial Image Dataset (AID) dataset [35] which has 10000 images with 30 different scene classes of size $600pt\times 600pt$. The 30 aerial scene types are as follows: airport, bare land, baseball field, beach, bridge, center, church, commercial, dense residential, desert, farmland, forest, industrial, meadow, medium residential, mountain, park, parking, playground, pond, port, railway station, resort, river, school, sparse residential, square, stadium, storage tanks and viaduct. Each class contains 220 to 420 samples. $N_{train}=8000$ samples are randomly selected as the training set with the other $N_{test}=2000$ samples as the test set. These two sets are consistent during training and testing stages of all aforementioned modules. The target modeled deployed on the back-end server is a ResNet34 with the input size of $224\times 224$. Therefore, all HR images are resized to this size. The size of each semantic block is $56\times 56$. The receptive field of proposed deep CS model compress is $k=16$. The sampling rate $sr$ is set to 0.3 for all of the semantic blocks. All experiments use 8-bit quantization in the compressed domain by default.

In the experimental environment set up in the previous section, the performance of two reward functions have been tested, as shown in Tab. I. The first two rows of Tab. I indicate 7 channel states and their corresponding transmission rates respectively. The third row means the transmission latency of full HR images in the compressed domain. The two reward functions are $\mathcal{R}_{1}=1/(1+2{T}),\eta_{1}=0.15$ and $\mathcal{R}_{2}=\exp(-2T),\eta_{2}=-0.02$. $\overline{N}^{a}=1/N_{test}\sum_{i=1}^{N_{test}}N_{i}^{a}$ represents the average number of semantic blocks extracted and transmitted by the two policies learned under different channel conditions. $\overline{B}=1/N_{test}\sum_{i=1}^{N_{test}}B_{i}$ indicates the average semantic data transmitted by the two policies under different channel conditions. $\overline{T}=\overline{B}/r_{D\rightarrow{B}}$ is the average transmission latency in these 7 states.

It can be seen from Tab. I that the proposed method can extract different amounts of blocks for transmission according to channel conditions. Additionally, the proposed policy learning method has been compared with 6 other policies under the premise of transmitting the same number of semantic blocks. Assuming that under the gain index $j\in\{1,2,3,4,5,6,7\}$, there are $N_{j}$ samples in the $N_{test}$ transmitted samples can be correctly classified by the back-end model by the proposed method, then its accuracy is calculated by $N_{j}/N_{test}$. Assuming that under the channel condition $j$, the number of semantic blocks transmitted by the $i$-th test sample is $N_{i,j}^{a}$, then other policies are controlled to also transmit $N_{i,j}^{a}$ semantic blocks, which can ensure the fairness of the experiment.

Now we will give a detailed introduction about several other comparative policies. $policy1$ selects semantic blocks from left to right in line order. $policy2$ extracts semantic blocks from top to bottom in the order of the columns. $policy3$ selects semantic blocks in a clockwise order from the center to the outside. On the contrary, $policy4$ extracts semantic blocks in a counterclockwise order from the center to the outside. $policy5$ randomly chooses $N_{i,j}$ semantic blocks that are the same as the proposed method for transmission each time. $policy6$ selects semantic blocks in order from largest to smallest in accordance with the saliency of semantic blocks. Tab. I reveals that the accuracy of proposed aerial image classification task offloading scheme is at least two percentage points higher than that of other traditional methods regardless of the channel conditions. Moreover, the proposed method can adaptively adjust the optimal policy of semantic extraction based on the change of channel conditions, focusing more on the semantic transmission latency when the channel conditions are poor while focusing on the completion of back-end classification task under better channel conditions.

Fig. 4 shows the number distribution of transmission blocks under different channel conditions with two different reward functions. The horizontal axis represents block numbers $N^{a}$ for transmission and the vertical axis represents number of samples that transmit $N^{a}$ semantic blocks ${Num}_{j}(N^{a})=\sum_{i=1}^{N_{test}}1_{\{N_{i,j}=N^{a}\}}$, where $1_{\{\cdot\}}$ indicates a 0-1 function being 1 when the condition in brackets is met, 0 otherwise. It is pretty obvious that the model tends to transmit more semantic blocks under high channel gain and tends to transmit fewer semantic blocks under low channel gain.

The back-end accuracies of target model with two reward functions compared to other policies under different channel conditions are shown in Fig. 5 and Fig. 6. The horizontal axis in each figure still shows 7 different channel gains, and the vertical axis denotes the classification accuracy of the back-end model under the constraints of $N^{a}$ transmission blocks. The number of samples successfully classified by the back-end model under the constraint of $N^{a}$ is ${Num}_{j}^{\prime}(N^{a})\sum_{i=1}^{N_{test}}1_{\{N_{i,j}=N^{a}\wedge\mathop{\arg\max}\limits_{k}f_{k}^{tar}=C\}}$. Therefore, the accuracy on the curves can be calculated as ${Num}_{j}^{\prime}(N^{a})/\sum_{m=1}^{N_{test}}1_{\{N_{i,j}=N^{a}\}}$. The two reward functions of proposed system can make the back-end accuracy better than other policies under the constraints of the same channel gain and transmission volume, especially in a poor channel environment.

Deep reconstruction of full HR images at the final stage are shown in Fig. 7, where from left to right are the original image, reconstructed full image with $sr=\{0.1,0.2,0.3,0.4\}$. Fig. 8 shows the policies learned with two reward functions. From left to right, they are the semantic blocks reconstructed under 7 channel conditions.