Communication Beyond Transmitting Bits: Semantics-Guided Source and Channel Coding

The technique of semantic guidance generation in our SCT embraces two functions, i.e., the semantic analysis transform and the semantic importance modeling. The semantic analysis transform is used for source semantic feature extraction and representation. Different from traditional linear transforms, such as the Karhunen-Loève transform (KLT), which map the source vector into a code space via a decorrelating invertible transform, the semantic analysis transform maps the source data to the semantic feature space via nonlinear transform operations that can much more closely match and represent the source semantic features [6].

With the right set of parameters, elaborate artificial neural networks (ANNs) with nonlinear properties can approximate arbitrary source distributions and extract the source semantic features. In our SCT system, the ANN-based semantic analysis transform produces the semantically annotated feature map in the semantic feature space. Taking the image source in Fig. 3 as an example, the semantic analysis transform module includes two functions:

• •

  Semantic feature extraction: It maps the original source to its corresponding feature map at the semantic feature space, which can be usually realized by convolutional neural networks (CNN), Transformer [7], etc. The feature map preserves the overall source semantic information that can be used to reconstruct the source data.

• •

  Semantic labeling: With semantic segmentation networks acting on the feature map, it produces the semantic label map delimiting multiple semantic regions. Combining it with the feature map, one gets the semantically annotated feature map.

The other critical function for generating semantic guidance is semantic importance modeling. It first generates the importance maps on the semantic feature space, then the resulting importance scores in terms of every SFV guides the following source-channel coding design. This function is highly tied with the communication purposes [8]. In particular, we consider two modes of semantic importance modeling, namely

• •

  Importance modeling toward human perception: For HTC scenarios, as humans can find different regions visually salient in a scene given the context, this mode generates the SFV importance scores combining entropy modeling [6] and saliency detection [9].

• •

  Importance modeling toward task accuracy: For MTC scenarios, as the gradients of task target can directly flow into the neural networks to highlight important regions in the feature map, this mode generates the SFV importance scores by jointly using entropy modeling [6] and gradient-weighted class activation mapping (Grad-CAM) [10].

We emphasize that for both modes, the entropy modeling is fundamental, which indicates the regional complexity in terms of the source data content. The human subjective perception or task attention is certainly a semantic refinement to the syntax entropy. As depicted in Fig. 3, the final importance map is a weighted sum over the entropy map and the saliency/activation map, where the specific weighting manner varies with scenarios. The semantic importance score of each SFV is obtained by counting up the importance values of feature map embeddings that belong to this SFV.

The receiver in SCT system invokes the semantic synthesis transform module to reconstruct the source data or to execute downstream tasks directly. As shown in Fig. 2, the semantic synthesis transform module mainly involves two functions, semantic distortion correction and semantic feature fusion, which are both nonlinear transforms. Explanations about semantic distortion correction will be given later since it plays a critical role in enhancing the fidelity of transmitted data in SCT. As for semantic feature fusion, it actually performs the inverse operation of the transmitter utilizing the corrected feature map to reconstruct the source data.

The aforementioned semantic guidance includes SFVs and their importance scores. It describes the source data semantics diversity, i.e., feature map embeddings and data samples are certainly not equally important. Similarity, wireless channels are of the channel diversity naturally, e.g., time and frequency selectivity in OFDM, spatial selectivity in MIMO, etc. Shannon theory focuses on how to well utilize the channel diversity to maximize the system capacity, which however discards the data diversity, considering data bits were born equal. For a long time, a question is how to exploit both types of diversity. Our SCT framework provides one solution by injecting semantic guidance into the design of source and channel coding. On the whole, source-channel coding plays a dual-functional role that compresses the input data while also combats the corruptions brought by wireless channels. In particular, as depicted in Fig. 3, our coding method includes two modes:

• •

  Overall preserving: This coding mode will be used when the semantic importance scores differ slightly. It preserves the overall source content to support the reconstruction of all regions with a high degree of detail at the receiver.

• •

  Selective preserving: This coding mode will be adopted when the semantic importance scores differ significantly. It completely generates parts of the source data from its semantic label map while preserving user-defined regions with a high degree of detail.

The overall preserving mode is often used in general HTC scenarios, where the communication purpose wants to preserve the full source data content as well as possible, while falling back to synthesized content instead of some visually-distorted regions for which not sufficient channel bandwidth is available to well transmit corresponding SFVs. The goal is high-fidelity data transmission to please human perception of all the source data contents. The selective preserving mode will be employed in most MTC scenarios and some specified HTC scenarios given the context. For example, in a human video call scenario [8] where the user wants to fully preserve people in the video stream, but a visually pleasing synthesized background from the semantic label map can serve the purpose as well as the true background. For both modes, our source-channel coding design obeys the following rules:

• •

  Heterogeneous transmitted data formats: For the overall preserving mode, sequences sent into source-channel encoders are all these SFVs. But for the selective preserving mode, our SCT system jointly transmits SFVs to reconstruct regions of interest and the semantic label map as tiny side information to synthesize other contents.

• •

  Differentiated coding rates: For SFVs with higher semantic importance scores, they will be allocated more coded symbols, and vice versa.

• •

  Matched channel resource assignment: For more important SFVs, their source-channel coded symbols will be transmitted over better conditioned channels, e.g., OFDM resource blocks (RBs) with higher channel gain.

Herein, source-channel coding can be implemented in either a modular way as that in traditional systems or an integrated way with one module, which will be introduced later. The first design rule indicates a heterogeneous coding formats under the selective preserving mode. Apart from the semantic critical regions transmitted by SFVs, other parts are alternated as the semantic label map as depicted in Fig. 3, which is formulated as a vector graphic. This amounts to a quite small and source dimension independent overhead in terms of source-channel coding cost. It contributes to a considerable overall reduction in channel bandwidth cost. In the receiver, after source-channel decoding, our SCT system seamlessly combines the preserved content which is generated from SFVs with the synthesized content which is generated from the semantic label map. This method can faithfully preserve the whole source semantics for MTC and some specified HTC scenarios.

As for the second and third rules, they are jointly considered to determine how many and which OFDM RBs are assigned to transmit source-channel coded symbols of each SFV. These RBs with higher channel gain will be of priority to be assigned to more important SFVs as shown in Fig. 3. In addition, the number of RBs assigned to an SFV is counted by aligning the RB sum channel capacity with the importance score of this SFV. For various scenarios, since the SFV importance scores are of different computation manners as aforementioned, the importance score is certainly of different meaning. Thus, the specific approach to map the importance score with the source-channel coding rate and channel resource assignment will be a critical issue to be optimized individually.

Following the aforementioned two techniques, the third key technique to improve the performance SCT system is semantic distortion correction. As shown in Fig. 2, it can exploit both intra- and inter-SFV correlations to further correct the residual errors left by source-channel decoding. This technique can be indeed categorized as one application of solutions for inverse problems in deep learning [11]. In other words, it operates on the representation in semantic feature space to recover the data in source space. As we should note, in our SCT system, semantic distortion correction is highly tied with previous two techniques. In particular, more efforts of distortion correction will be put on these less important SFVs, which often incurs more left errors due to the less allocated channel transmission resources. Fortunately, they are usually easy to be restored due to the strong side information provided by the well transmitted SFVs of higher importance. This process is quite different from traditional post-processing tools used to correct residual errors after source-channel decoding. In that case, the residual error distribution is not relevant to the source semantics. Thus, it will be inefficient and ineffective to restore these distortions due to the lack of semantic guidance generation and the biased coding strategy.

Specifically, the semantic distortion correction technique can be realized by generative neural networks, such as the generative adversarial network (GAN), which are of “creativity” by exploiting the semantic-level correlations to generate distortion parts of the source content. These deep learning methods have been studied widely in solving inverse problems, such as image post-editing, inpainting, denoising [12], etc. Herein, it is a new application in communications. A detailed procedure is demonstrated in Fig. 4, where the goal of SCT is to reconstruct image. The overall structure exploits conditional GANs which introduce additional information into GANs as a conditional model. This additional information can be the semantic label map, error masks, local reference images, or even class labels, etc. The distortion correction network restores the erroneous feature map according to these conditions and then feeds it to the generator (semantic feature fusion module) to reconstruct the source image.

We emphasize that the semantic distortion correction technique often needs some additional side information as conditions. Partial side information needs to be transmitted, such as the semantic label map formulated as a simple vector graphic [8], while some other side information like references can be stored locally as knowledge base. However, the coded transmission costs of communicated side information are usually marginal compared to that in the primary data link to transmit necessary SFVs. This almost-negligible overhead exchanges a considerable system performance improvement, which will finally contributes to the reduction in the number of automatic repeat requests (ARQs) that leads to a higher link throughput and lower latency. On the whole, it is worth to transmit the marginal side information supporting the semantic distortion correction.

As a compatible scheme with current digital communications systems (e.g., 4G and 5G), the modular implementation of SCT retains channel coding and digital modulation modules to transmit digital signals.

As an example, consider an image source in Fig. 2, the semantic guidance generation module produces the semantically annotated feature map first, which is then divided into multiple SFVs. At the step of source-channel encoding, every SFV is first quantized and interleaved, then compressed by an entropy encoder into bits, and finally loaded into packages. Advanced channel coding, such as low-density parity-check (LDPC) or polar codes, follows to protect the source-coded bits against channel corruptions. The overall source-channel coding rate of each SFV is assigned according to its semantic importance score. In the receiver, the digital demodulation operates on the received signal first, it outputs the soft estimations of digit bits which are sent into their corresponding channel decoder. The channel decoded messages are then fed into the source decoder to reconstruct SFVs. A major disadvantage of this approach is that residual errors in channel decoded messages will incur severe error propagation in entropy decoding. However, in our SCT system, this undesirable phenomenon can be alleviated by the semantic distortion correction module. By generating the corrupted regions according to semantic description and context information, the end-to-end transmission quality can be improved.

Fig. 5 demonstrates an example of modular implementation of SCT for image transmission. Here, a 2048 (height) $\times$ 1024 (width) cityscapes image [13] is transmitted to the receiver over the additive white Gaussian noise (AWGN) channel with signal-to-noise ratio (SNR) 1dB. Among multiple semantic regions in the cityscapes, humans, vehicles, and buildings are assessed of higher importance, but other natural environments, e.g., sky and trees, are of lower importance. For the classical coded transmission, we plot the reconstructed image of the concatenation of BPG codec followed by an LDPC channel codec. As we can see, it fails to reconstruct all regions due to the residual bit errors left by the channel decoder propagating among other coded bits. For our SCT system, residual channel decoded bit errors also lead to a corrupted semantic feature map (plotted in the left top, where the black pixels indicate the corrupted locations), then cause the mosaics during the reconstruction process. The main difference between classical coded transmission and SCT is that our semantics guides the source-channel coding design to match with the importance score of each SFV. In this way, errors tend to cause corruption in SFVs with lower importance. As shown in the right sub-figure, such corruptions can be further corrected by using an inpainting model [12] so that the image quality increases visibly.

Following the joint source-channel coding idea, the source-channel coding part in the SCT system can also be designed as an integrated module, which does not rely on explicit codes for either compression or error correction; Instead, it maps SFVs to channel input analog symbols directly. The encoder and decoder functions are parameterized as two jointly trained ANNs. They together constitute an autoencoder affected by a non-trainable layer that represents the wireless communication channel [14]. The joint coding rate of each SFV is aligned with its semantic importance score. The learned source-channel coding allows gradients propagating between coding modules and semantics guidance generation modules. This integrated implementation manner can thus achieve end-to-end learning to minimize the perceptual loss or handle the downstream tasks directly. The whole procedure is also depicted in Fig. 2.

We exemplify the SCT performance based on the integrated implementation and analog transmitted signals. For a general HTC scenario where images are transmitted over wireless links to meet human browsing requirements, to quantify the effectiveness of SCT, we adopt the emerging learned perceptual image patch similarity (LPIPS) metric to indicate the human perception [15]. Fig. 6 presents the quantitative comparison of transmission rate-distortion (RD) performance over three datasets of different resolutions. The distortion $D$ is quantified as the LPIPS loss (lower is better), and the transmission rate is defined as the channel bandwidth ratio $R$ (defined as the number of transmitted symbols per pixel). As comparison, we provide the performance of BPG compression combined with the ideal capacity-achieving channel code family to serve as a bound of the traditional coded transmission [14]. We find that the integrated SCT can well surpass the traditional scheme especially in the low rate region ($R<0.6$). In the high rate region, traditional coding performs better due to its elaborate design. Even so, as an emerging scheme, our SCT has shown its potential, which can be further improved by using more advanced neural network architectures and elaborate designs.

In contrast to traditional coded transmission under the umbrella of Shannon information theory, the SCT system based on deep learning methods lack solid mathematical foundations for theoretical analysis. Consequently, errors may not be well-bounded when the semantic nonlinear transform and semantic distortion correction techniques (such as GANs) are used. Due to this, the transmission rate of SCT may not be well-bounded either, nevertheless, this is desired to guide the practical design of SCT and gain more insights into the performance limits.

A potential solution to address this challenge is expanding the concept of typical sequences in the Shannon information theory with respect to the semantic aspects. By performing reasonable merging operations on the typical sequences belonging to the same semantics, one may try to analyze the asymptotic performance of SCT. In addition, the Kolmogorov complexity, defined as the shortest computer program describing the source sequence, may be used to quantify the rate of semantic compression and analyze the tradeoff function between the channel transmission rate and the end-to-end semantic distortion, i.e., the semantic-level RD function.