Multi-Modal Fusion-Based Multi-Task Semantic Communication System

According to Weaver[11], communications can be categorized into three levels: the technical level, which ensures the accuracy of bit sequence transmission; the semantic level, which ensures transmitted symbols accurately convey the desired meaning; and the effectiveness level, which ensures the received meaning affect conduct in the desired way. In the contemporary field of wireless communications, data rates are approaching Shannon’s capacity limit. Traditional data transmission methods established at the technical level often require high communication overhead, especially when dealing with large datasets or multi-modal data[12]. Therefore, we need a more efficient communication method. Building upon Weaver’s seminal categorization[11], many studies [5, 2, 6, 7] are dedicated to studying a new paradigm at the semantic level, namely semantic communications.

Semantic communication systems show great promise. They interpret information at the semantic level rather than mere bit sequences[2], which aims to achieve more intelligent, efficient, and reliable data transmission. The benefits of semantic communications mainly lie in reducing communication overhead and improving downstream task performance. They eliminate redundant content and focus on the meaningful information for downstream tasks, thus preventing the impact of irrelevant information on task performance. On the other hand, by transmitting task-related information, the bandwidth and communication latency are reduced.

The opening work on semantic communications is DeepSC[2], which is developed for text reconstruction. This research applies Transformer[13] to communications. Due to the self-attention mechanism, the performance of DeepSC far exceed traditional communication methods. As more and more semantic communication models are proposed[5, 6, 7], semantic communications have expanded beyond text transmission and are now active in the transmission of other modal data such as speech, image, and video.

However, existing works mainly concentrate on single-task or single-modal scenarios. Due to the growing diversity of data, information is no longer limited to a single format or target. Image, text, speech, and video data often coexist, and we need to use them together to perform various tasks. Moreover, current research often overlooks the correlations among multi-modal data. This reduces the ability of models to understand tasks. By solving the challenges posed by multiple modalities and multiple tasks, communication systems can represent semantic information more deeply and accurately. In this paper, we delve into these shortcomings, aiming to bridge the gap between existing semantic communication research and the demands of real-world applications, thereby creating a more comprehensive and effective multi-modal and multi-task semantic communication framework.

Recent research has made significant strides in multi-task models[14, 15]. Various techniques like soft sharing[16] and hard sharing[17] have been explored to enhance the performance of individual tasks through cross-task knowledge transfer. Hard sharing is the most widely used sharing mechanism. It embeds the data representations of multiple tasks into the same semantic space, and then uses a task-specific layer to extract task-specific representations. Hard sharing is suitable for processing tasks with strong correlations, but it often performs poorly when encountering tasks that are less related[18]. Soft sharing builds a unique neural network for each task. It is suitable for situations where the correlations among tasks are not strong[18] but increases storage overhead. In addition to the inherent shortcomings, these approaches often face challenges when applied to tasks involving multi-modal data.

In multi-modal tasks, the focus is on leveraging complementary information from different data modalities. Fusion techniques such as early fusion, intermediate fusion, and late fusion have shown promise[19]. Many researchers use architectures such as Multilayer Perceptron (MLP) and Convolutional Neural Network (CNN)[19] in multi-modal fusion, which has indeed made some progress. Nevertheless, these methods are difficult to adapt to the changing relationships among different data[20]. With the deepening of research, the attention mechanism[13] has gradually attracted researchers.

In various fields, the attention mechanism has shown excellent results. The Transformer[13] model improves the ability of text understanding, and models such as Vision Transformer[21], Speech Transformer[22], and Video Vision Transformer[23] introduce self-attention mechanism into image, speech and video processing, significantly improving the performance of corresponding tasks. These studies show that the attention mechanism exhibits outstanding performance and application potential, whether in different tasks or in different modal data.

Considering that tasks with the same modal data are usually more closely related, while tasks with different data modalities are relatively weakly related. Related tasks often have inter-dependence and perform better when solved in a joint framework[24]. Therefore, we propose to use a combination of hard sharing and soft sharing. We design independent semantic encoders for different modalities while utilizing shared channel encoder and channel decoder to achieve unified compression and recovery. In order to prevent negative transfer among unrelated tasks, we add additional task embeddings[25] for each task to enhance the distinction. To fully exploit the complementarity of multi-modal data and reduce the cost of transmission, we propose an innovative fusion module based on BERT[26] to fuse multi-modal semantic information. In multi-modal tasks, the semantic features of each modality are extracted by corresponding semantic encoders. These features are concatenated as input sequences which are then semantically fused by the fusion module. This exploration improves the task performance and provides a new paradigm for multi-modal tasks in semantic communications.

Unlike traditional approaches that usually deal with single-modal data and single-task data, we consider multi-modal and multi-task scenarios that require processing different data modalities and performing multiple tasks simultaneously. Our multi-modal fusion-based multi-task semantic communication system is shown in Fig. 1. The system encompasses a task set comprising numerous tasks, each contains various modal data. It is designed to support four modalities: image, text, speech, and video. In the system, three components play essential roles: transmitter, physical channel, and receiver.

The transmitter consists of semantic encoder and channel encoder. In the semantic encoder, we build four distinct encoders, a text encoder, an image encoder, a speech encoder, and a video encoder. Each encoder is tailored to one modality. This architecture can better extract semantic information from different modal data. In addition, we design a BERT-based fusion module to fuse multi-modal semantic information. It takes the concatenation of different modal semantic information as input sequences, and further uses the self-attention mechanism to fuse them. This module enhances the processing capabilities for multi-modal tasks and significantly reduces the cost of data transmission.

The primary function of the channel encoder is to compress the high-dimensional semantic information obtained from the semantic encoder. Utilizing neural network techniques, the channel encoder can effectively reduce the dimension of the data. This reduction eases bandwidth demands while maintaining the semantic integrity of the original semantic information.

The compressed semantic information will be transmitted to the physical channel. We mainly consider two channels, the Additive White Gaussian Noise (AWGN) channel and the Rayleigh fading channel. Both of them represent typical wireless channel environments. The AWGN channel is widely used to simulate an ideal wireless environment, and the Rayleigh fading channel is more suitable for modeling complex wireless environments, such as heavily built-up urban environments. By studying both wireless transmission environments, we can fully evaluate the performance of our system.

The semantic information reaches the receiver after passing through the physical channel. The receiver incorporates channel decoder and semantic decoder. The channel decoder based on deep learning is a key component of the semantic recovery. Its main responsibility is to reconstruct undistorted high-dimensional semantic information. The recovered semantic information is fed into the semantic decoder, which consists of a series of lite neural network task heads. Each task head is optimized for a specific task. The use of lite task heads can ensure fast response of the entire system.

In this work, we consider $K$ tasks, denoted by a set $\mathcal{K}=\{1,2,\cdots,K\}$. In the task set, there are multiple tasks $T=\{T_{1},T_{2},...,T_{K}\}$, where $T_{i}$ represents the $i$-th task and $i\in\mathcal{K}$. The modality of source data $D_{i}$ in $T_{i}$ is $M_{i}=\{M_{1},M_{2},...,M_{m}\}$. We consider four modalities, namely $M_{i}\subseteq\{\mathcal{I}:image,\mathcal{T}:text,\mathcal{V}:video,\mathcal{S}:speech\}$.

We employ Deep Neural Network (DNN) to design both the transmitter and receiver. The transmitter consists of two parts, the semantic encoder and the channel encoder. They are designed to extract semantic information from $D_{i}$ and ensure the successful transmission over the physical channel. For task $T_{i}$, the encoded symbol stream $X_{i}$ can be represented as

where $S(\cdot;\alpha_{T})$ is the semantic encoder network with the parameter set $\alpha_{T}$ and $C(\cdot;\beta_{T})$ is the channel encoder network with the parameter set $\beta_{T}$. Then $X_{i}$ is transmitted over the physical channel. This process can be modeled as

where $H$ represents the channel matrix and $N\sim\mathcal{CN}(0,\sigma^{2}I)$ is the Gaussian noise.

The semantic information is sent to the receiver after passing through the physical channel. The channel decoder first restore semantic information and the semantic decoder processes the recovered semantic information to perform intelligent tasks. This process can be denoted as

where $S^{-1}(\cdot;\alpha_{R})$ is the semantic decoder network with the parameter set $\alpha_{R}$ and $C^{-1}(\cdot;\beta_{R})$ is the channel decoder network with the parameter set $\beta_{R}$. $\hat{Y_{i}}$ is the predicted result of task $T_{i}$.

For task $T_{i}$, its performance metric function is given as $P_{i}(\hat{Y_{i}},Y_{i})$, where $Y_{i}$ represents the ground truth. Our goal is to maximize the metric function for every task

In order to maximize the metric function of each task, it actually means that we need to minimize the loss function of them, which is given by

$L_{i}$ represents the loss function of $T_{i}$.

We strive to design a comprehensive semantic communication system to meet this objective. The system can adapt to the needs of different tasks and take advantage of the complementarity among various data modalities.

To comprehensively analyze the system, we consider single-modal tasks of image, text, speech, and video as well as some multi-modal tasks.

In our multi-modal fusion-based multi-task semantic communication framework, the semantic encoder plays a crucial role. In the design of semantic encoder, our innovation is mainly reflected in two places. First, we design specialized semantic encoders for different modalities, and each encoder aims to effectively extract corresponding semantic features. This is because we need to take into account the heterogeneity of data from different modalities. Using a single encoder for all modalities will ignore specific information. By designing dedicated encoders for each modal data, the framework can adapt to the complexity of different modalities. Tasks with the same modal data can also enhance knowledge sharing by using the same encoders. Second, considering the complementarity and redundancy of different modal semantic information in multi-modal tasks, we design a fusion module. Information across various modalities is typically complementary, with each providing distinct details. Through the fusion module, we can effectively integrate the semantic information of these different modalities to generate a more comprehensive feature representation. In addition, through multi-modal fusion, the amount of data in the communication can be significantly reduced. In the following, we detail the semantic encoder corresponding to each modality.

The semantic encoder we design for each modality takes into account the characteristics of each modal data. It aims at extracting distinct semantic features. For multi-modal tasks, semantic features of different modalities provide different details, and fusing them can improve the performance of the model while reducing redundancy. However, due to the heterogeneity of different modal data, these semantic features are not in the same semantic space. This means that neglecting multi-modal fusion and simply concatting them together will cause misalignment of semantic information. To solve this problem, we design a fusion module based on BERT[26], which is depicted in Fig. 6. The key of this method is to use the self-attention mechanism to achieve fusion of multi-modal semantic features. The self-attention mechanism is adept at capturing the interdependencies among different modalities. It dynamically weights the importance of each modal semantic features, effectively realigning them into a unified semantic space. This enables the model to leverage complementary information.

Assume that $\mathcal{D}=\{(D^{M_{1}},D^{M_{2}},...,D^{M_{m}},Y)\}$ is a multi-modal dataset, where $D^{M_{m}}$ and $Y$ are the source data with $M_{m}$ modality and its corresponding labels, respectively. Input the data into the corresponding semantic encoder, we can obtain the corresponding semantic features. These semantic features are represented as $F^{M_{1}}\in\mathbb{R}^{L_{M_{1}}\times P},F^{M_{2}}\in\mathbb{R}^{L_{M_{2}}\times P},...,F^{M_{m}}\in\mathbb{R}^{L_{M_{m}}\times P}$. In this case, we regard the semantic features extracted from each modality as sequence features. $L_{M_{j}}$ is the length of the sequence and $P$ is the dimension of feature vecotors, where $j$ is between $1$ and $m$. Subsequently, these features are sent to the fusion module for fusion. The steps for our fusion module to fuse different modal semantic features are as follows.

• •

  Step 1 (Concatenation): First, we need to concat semantic features of different modalities. We can get the concatenation of semantic features given by

  $$F^{C}=\mbox{concat}(F^{M_{1}},F^{M_{2}},...,F^{M_{m}})$$

  (9)

  where $F^{C}\in\mathbb{R}^{L^{\prime}\times P}$ and $L^{\prime}=\sum_{j=1}^{m}L_{M_{j}}$.

• •

  Step 2 (Segment Embeddings): Unlike the traditional Transformer models, our fusion module does not incorporate position embeddings. This is because position embeddings have been added in the text, speech, and video semantic encoder. And image semantic features are essentially the feature map, so there is no need for position embeddings. Moreover, for the concatenated features, we think they are all equally important and the sequence order should have no effect on the task performance. Therefore, we only consider segment embeddings. In this context, we treat different modalities of semantic features as distinct segments. Each modal $M_{j}$ has a unique segment embedding vector $E_{s}^{M_{j}}\in\mathbb{R}^{1\times P}$. For each sequence vector in $F^{C}$, we select the corresponding segment embedding vector. All segment embedding vectors are concatenated together to get the segment embeddings $E_{s}\in\mathbb{R}^{L^{\prime}\times P}$. Then, the input embeddings are denoted as

  $$F^{in}=F^{C}+E_{s},\quad F^{in}\in\mathbb{R}^{L^{\prime}\times P}.$$

  (10)

  Through segment embeddings, the distinction of semantic features of different modalities can be enhanced, allowing our fusion module to better capture the distinct semantic features of each modality and cross-modal interdependencies. In addition to segment embeddings, we concat the task embedding vector and $F^{in}$ to prevent negative transfer of effects. Therefore, the dimension of $F^{in}$ are $\mathbb{R}^{L\times P}$, where $L=L^{\prime}+1$.

• •

  Step 3 (Attention Layer): $F^{in}$ is then input into multiple attention layers. We design 6 attention layers in our fusion module. Each layer consists of two parts: multi-head self-attention (MSA) and feedforward neural network (FFN). Suppose that $H^{(l)}\in\mathbb{R}^{L\times P}$ is the output by the $l$-th attention layer, and $H^{(0)}$ is $F^{in}$. Then $H^{(l)}$ can be denoted as

  	$\displaystyle H^{(l)^{\prime}}$	$\displaystyle=\mbox{LN}(\text{MSA}(H^{(l-1)})+H^{(l-1)})$		(11)
  	$\displaystyle H^{(l)}$	$\displaystyle=\mbox{LN}(\text{FFN}(H^{(l)^{\prime}})+H^{(l)^{\prime}})$

  where LN is the layer normalization and $H^{(l)^{\prime}}$ is the intermediate features of $l$-th layer that passes through MSA. MSA mechanism allows the fusion module to focus on different parts of input sequences, enabling it to capture a wide range of dependencies. This is given by

  $$\text{MSA}(H^{(l-1)})=\text{concat}(\text{head}_{1},...,\text{head}_{h})W^{O}.$$

  (12)

  $W^{O}\in\mathbb{R}^{P\times P}$ is a learnable parameter matrix and $\text{head}_{i}\in\mathbb{R}^{L\times P_{h}}$ is calculated by self-attention (SA). $P_{h}$ is the dimension of each head. In our work, we consider 12 heads, and $P_{h}=P/12$. For each head output by SA, it can be represented as

  	$\displaystyle\text{head}_{i}$	$\displaystyle=\text{SA}(Q,K,V)$		(13)
  		$\displaystyle=\text{softmax}(\frac{(QW_{i}^{Q})(KW_{i}^{K})^{T}}{\sqrt{P}})VW_{i}^{V}$

  where $W_{i}^{Q}\in\mathbb{R}^{P\times P_{h}}$, $W_{i}^{K}\in\mathbb{R}^{P\times P_{h}}$, $W_{i}^{V}\in\mathbb{R}^{P\times P_{h}}$ are weight matrices for the query $Q\in\mathbb{R}^{L\times P}$, key $K\in\mathbb{R}^{L\times P}$ and value $V\in\mathbb{R}^{L\times P}$, respectively. Since we use self-attention mechanism, $Q$, $K$, $V$ are actual is $H^{(l-1)}$. In softmax, we added a scaling factor $\sqrt{P}$, which is to prevent the gradient vanishing problem caused by excessive attention value. The output of MSA is then processed by the FFN, which applies two linear layers with the GeLU activation function:

  $$\mbox{FFN}(H^{(l)^{\prime}})=\mbox{GeLU}(H^{(l)^{\prime}}W_{1}+b_{1})W_{2}+b_{2}$$

  (14)

  where $W_{1},b_{1},W_{2},b_{2}$ are parameters of the two linear layers.

• •

  Step 4 (Average Aggregation): After six attention layers, we derive the features $H^{(6)}\in\mathbb{R}^{L\times P}$. Then, we perform an average aggregation on the first dimension to obtain a fused feature vector $F^{f}\in\mathbb{R}^{1\times P}$. The fused semantic features then are sent to channel encoder. For single-modal tasks, the semantic features do not go through the fusion module, but are directly dispatched to the channel encoder.

Semantic features from differenct modalities often face misalignment issues in multi-modal fusion due to discrepancies in semantic spaces. Employing the attention mechanism enables the fusion module to align these varied semantic features, enhancing multi-modal task performance while reducing transmission redundancy.

In the channel encoder, we use two layers of Multilayer Perceptron (MLP), which is shown in Fig. 7. The first MLP layer is used to receive the original semantic features and compress them. The second MLP layer is used to further compress the output of the first layer. This layer has fewer neurons, resulting in a lower output dimension compared to the first layer. In this way, the network can learn more abstract feature representations. The output of the second MLP layer can be considered a compact representation of the original semantic features, with higher information density, while further reducing communication overhead of semantic communication.

The compressed semantic features contain the core information of the original features but have lower dimensions. To restore the information, we also use two layers of MLP for channel decoding, which is depicted in Fig. 8. The first MLP layer restores the information of the original features. The second MLP layer is used to further improve the decoding process. This layer has more neurons, enabling it to learn more complex feature reconstruction patterns.

We choose a simplified and efficient approach, avoiding complex neural networks as the semantic decoder. It is demonstrated in Fig. 9. We design specific lite neural networks for each task head which only contains one or two linear layers. This simplified design is based on the following considerations: First, we find that a complex architecture does not necessarily bring significant improvements in performance. Instead, using simple linear layers, sufficient accuracy requirements can be achieved. Furthermore, the simple linear layer architecture allows us to significantly reduce the consumption of computing resources without sacrificing too much accuracy. This design makes our framework excel in flexibility.

We conduct joint training for all tasks. Each time, we randomly select one task for training. During the training process, we employ the Adam[31] optimizer for parameter optimization. Specifically, we create a separate Adam optimizer for each task. This allows the model to adjust the learning rate and parameter updates for each task more effectively. The whole process is shown in Algorithm 1. During this process, the semantic encoder $S(\cdot;\alpha^{M_{j}}_{T})$ of each modality $M_{j}$, fusion module $F(\cdot;\gamma)$, channel encoder $C(\cdot;\beta_{T})$, channel decoder $C^{-1}(\cdot;\beta_{R})$, and semantic decoder $S^{-1}(\cdot;\beta_{R})$ will be optimized. We evaluate the performance of the previously trained model in AWGN channel and Rayleigh fading channel under various SNR (signal-to-noise) conditions. The whole process is shown in Algorithm 2.

To avoid the conflits of different tasks, we set an Adam optimizer for each task. Except for the speech recognition task, where the learning rate is 2e-4, the learning rates of other tasks are set to 1e-4. To better verify the effectiveness of MFMSC, we include multiple baselines and existing models for comparison.

We conduct extensive evaluation on multiple tasks. Fig. 10 illustrates the performance of our proposed framework under AWGN channel. Traditional communication methods perform poorly under low SNR conditions, while those semantic communication models show good performance. Each of them demonstrates the ability of maintaining the semantic integrity against the backdrop of noise. As the SNR increases to higher levels, the MFMSC stands out, often achieving the highest performance. Furthermore, the performance of our proposed MFMSC is close to MFSC. This shows that MFMSC has good support for multi-task semantic communication systems. Moreover, we extend the test to Rayleigh fading channel conditions, which is depicted in Fig. 11. Since we pay more attention to multi-modal tasks, only the performance of VQA v2 and MM-IMDb is shown in this figure. MFMSC once again demonstrates strong performance, showing no significant performance degradation. The findings indicate that our framework effectively extracts the semantic information, showcasing its application potential.

In order to better compare the effect of our multi-modal fusion, we set up a baseline model MMSC. Our experimental results are demonstrated in the Fig. 10(g), Fig. 10(h), Fig. 11(a) and Fig. 11(b). Whether it is the AWGN channel or the Rayleigh fading channel, the performance of MFMSC is significantly stronger than that of MMSC. This is beacuse that MMSC lacks the fusion module and often fail to capture the complementary features of different modalities. The drawback results in reduced task performance and higher communication overhead. Compared with the MMSC, MFMSC successfully achieves data fusion among different modalities through the fusion module. In addition, for single-modal tasks, we can find that the fusion module has almost no impact on the performance of them. From Fig. 10, it is apparent that the performance of MFMSC and MMSC is essentially on par in single-modal tasks. We extend our study analysis to T-DeepSC and U-DeepSC. These two models that similarly do not take advantage of modal fusion in their architectures. Like MMSC, U-DeepSC and T-DeepSC concat multi-modal semantic features together and then transmit them into the physical channel. We find that their performance is also significantly inferior to our MFMSC model. Clearly, the lack of fusion mechanism limits their ability to exploit complementary features. Our experiments demonstrate that MFMSC greatly enhances multi-modal task outcomes. And this is due to our BERT-based fusion module. This module uses the self-attention mechanism to align and fuse the semantic features in different semantic spaces of each modality, thereby realizing the interaction of modal information and making full use of the complementary information among modalities.

For semantic communication, the evaluation of task performance is important, but equally important is the communication overhead. In practical applications, especially when resources are limited, communication overhead often determine whether a system has practical value. Considering that multi-modal tasks will bring additional communication overhead, in this work, we compare the communication overhead between the MFMSC and other models in multi-modal tasks.

In MFMSC, the fusion module plays a great role in eliminating redundancy. For the multi-modal data, each modality $M_{j}$ is extracted semantic features through a semantic encoder and the dimension is represented as $\mathbb{R}^{{L_{M_{j}}}\times P}$. ${L_{M_{j}}}$ represents the sequence length of modal $M_{j}$, and $P$ represents the feature dimension. Then we concat them and task embedding vecotor together. The concatenation is sent to the fusion module. Assuming that there is $m$ modalities in total, then the dimension of concatenated semantic features is $\mathbb{R}^{L\times P}$, where $L=\sum_{j=1}^{m}L_{M_{j}}+1$. We fuse these modalities into a more compact representation, and the dimension of fused data is $\mathbb{R}^{1\times P}$. Therefore, compared with MMSC, we can reduce the amount of transmitted data to $1/L$ of the original amount through multi-modal fusion.

We calculate the amount of data transmitted per task instance in the VQA v2 and MM-IMDb datasets, which is shown in Table I. Compared with MMSC, T-DeepSC and U-DeepSC, our MFMSC reduces the communication overhead by 98.0%, 46.7%, and 46.7% on the VQA task, respectively. On the MM-IMDb task, the communication overhead are reduced by 96.8%, 63.6%, and 63.6%, respectively. This shows that MFMSC can reduce the communication overhead while improving multi-modal task performance, giving it better application prospects.