Semantic Communication with Memory

We assume the $k$-th context, ${\bm{s}}(k)$, is transmitted during the $k$-th time slot and denote ${\bm{s}}^{c}$ and ${\bm{s}}^{q}$ as the context sentence and question sentence, respectively. In the memory shaping phase, the transmitter sends the context, e.g., multiple sentences, images, or speeches, to the receiver over multiple time slots. Subsequently, with the semantic encoder and channel encoder, the $k$-th context sentence over the $k$-th time slot can be encoded as

where ${\bm{x}}^{c}(k)\in\mathbb{C}^{L\times 1}$ is the transmitted signals after the power normalization, $S\left(\cdot;{\bm{\alpha}}\right)$ and $C\left(\cdot;{\bm{\beta}}\right)$ are denoted as the semantic encoder with parameter ${\bm{\alpha}}$ and channel encoder with parameter ${\bm{\beta}}$, respectively.

Transmitting the signals over the channels, the received signal can be presented as

where ${\bm{h}}(k)$ is the channel coefficients and ${\bm{n}}$ is the additive white Gaussian noise (AWGN), in which ${\bm{n}}(k)\sim\mathcal{CN}\left(0,\sigma_{n}^{2}{\bf I}_{L}\right)$. For the Rayleigh fading channel, the channel coefficient follows ${\bm{h}}(k)\sim{\cal CN}\left(0,{\bf I}_{L}\right)$; for the Rician fading channel, it follows ${{\bm{h}}(k)\sim\cal CN}\left(\mu_{h}{\bf I}_{L\times 1},\sigma_{h}^{2}{\bf I}_{L}\right)$ with $\mu_{h}=\sqrt{r/(r+1)}$ and $\sigma_{h}=\sqrt{1/(r+1)}$, where $r$ is the Rician coefficient. The SNR is defined as $\mathbb{E}({{{\left\|{\bm{h}}(k)\odot{\bm{x}}^{c}(k)\right\|}^{2}}})/\mathbb{E}({{{\left\|\bm{n}(k)\right\|}^{2}}})$.

With the estimated channel state information (CSI), $\hat{\bm{h}}$, the transmitted signals, $\hat{{\bm{x}}}(k)$, can be detected by

After signal detection, the semantic features can be recovered by

where $\hat{\bm{z}}^{c}{(k)}\in\mathbb{R}^{N\times 1}$ and $C^{-1}\left(\cdot;{\bm{\gamma}}\right)$ is denoted as the channel decoder with parameter ${\bm{\gamma}}$. Then, the recovered semantic features will be inputted into the memory module.

Inspired by the short-term memory, we model the memory module as a queue with length $T$, which only concerns the context. The memory queue at the $k$-th time slot is represented by

where ${\mathbf{Q}{(k)}}\in\mathbb{R}^{N\times T}$, $T$ is the length of memory. For $k<0$, $\hat{\bm{z}}^{c}{(k)}$ is the vector with all zero elements. The memory queue is updated with the incoming received latest semantic features and pop the oldest features out of the queue. For example, the memory queue at the $(k+1)$-th time slot is given by

For scenario communications, it is important to recognize the sequence of the memory queue. In other words, we need to know the order of sentences happened earlier or later. Therefore, we need to add the temporal information before inputting to the models, which is given by

where $\mathbf{T}=[{\bm{t}}_{1};{\bm{t}}_{2};\cdots;{\bm{t}}_{T}]\in\mathbb{R}^{N\times T}$ is the temporal information matrix, any two elements of which satisfy $<{\bm{t}}_{m},{\bm{t}}_{n}>=g(m-n)$ for some function $g(\cdot)$. We choose the positional coding employed in Transformer [27] here.

When performing the memory task, the memory shaping phase will introduce the overheads in terms of time availability due to occupying multiple time slots. For example, considering the task offloading scenario, compared to the memoryless tasks that only use several time slots, the memory shaping will occupy multiple time slots and may block the transmission pipe. This makes the time consumption a little high at the initial memory shaping phase. However, when the memory module is shaped, such consumption can be alleviated by reusing or partly updating the shaped memory module, because the memory module serves multiple questions. Therefore, the frequency of carrying memory shaping phase depends on the demands of users. When performing the memory task, the user will decide whether the current context stored in the memory module can answer the question. If not, the memory needs to be updated.

In the task execution phase, the transmitter sends the question sentence, ${\bm{s}}^{q}$, to the receiver to perform the task. Specially, ${\bm{s}}^{q}$ is encoded into ${\bm{x}}^{q}$ by (1), transmitted over the air, and decoded into $\hat{\bm{z}}^{q}$ by (4). In the scenario question answer task, the question is not only relevant to only one context sentence but also multiple context sentences. Therefore, the answer is predicted with the question and memory together, which is represented as

where $S^{-1}(\cdot;{\bm{\varphi}})$ is the semantic decoder with parameters ${\bm{\varphi}}$.

The proposed Mem-DeepSC is shown in Fig. 2. The semantic encoder consists of universal Transformer encoder layer with variable steps to extract the semantic feature of each word. In order to reduce the transmission overheads, the summation operation is taken here, in which these semantic features at the word level are merged to get one semantic feature at the sentence level. The reason that we choose universal Transformer in the semantic codec can be summarized as follows:

• •

  The state-of-arts, i.e., BERT [28] and GPT-3 [29], choose Transformer to deal with textual information, which shows the powerful extraction capability of text semantic information.

• •

  The universal Transformer can be trained and tested much faster than the architectures based on recurrent layers due to the parallel computation [26].

• •

  Compared with the classic Transformer, the universal Transformer shares the parameters, which can reduce the model size.

After the semantic encoder, the JSC encoder employs multiple dense layers to compress the sentence semantic feature. The reasons that we mainly use dense layer in the channel codec can be summarized as follows:

• •

  The JSC codec aims to compress the semantic features and transmit it effectively over the air. Compared with the CNN layer to capture the local information, the dense layer is good at capturing the global information and preserving the entire attributes, which follows the target of the JSC codec. This can enhance the system’s robustness to channel noise.

At the receiver, the JSC decoder correspondingly includes multiple dense layers to decompress sentence semantic feature and reduce the distortion from channels. The semantic decoder also contains the universal Transformer encoder layer with variable steps to find the relationship between the memory queue and the query feature to get the answer. Especially, the memory queue does not contain the temporal information inside. Therefore, temporal coding is employed to add temporal information to the memory queue, in which we adopt the positional coding [27] as the temporal coding.

As shown in Algorithm 1, the training of Mem-DeepSC includes three steps, which is similar to the training algorithm proposed in [17]. The first step is to train the semantic codec. In order to improve the accuracy of answers, we choose the cross-entropy (CE) as the loss function instead of the answer accuracy. The cross-entropy is given by

where $p\left(a\right)$ is the real probability of answer and $p\left(\hat{a}\right)$ is the predicted probability. After convergence, the model learns to extract the semantic features and predict the answers. The following proposition proven in Appendix A reveals the relationship between cross-entropy and the answer accuracy.

After converged, the model is capable of extracting semantic features and predicting the answer accurately. Subsequently, the second step is to ensure the semantic features transmitted over the air effectively. Thus, the JSC codec is trained to learn the compression and decompression of the semantic features as well as to deal with the channel distortion with the mean-squared error (MSE) loss function,

where ${\bm{z}}^{c}(k)$ and $\hat{\bm{z}}^{c}(k)$ are the original semantic features and the recovered semantic features, respectively.

Finally, the third step is to optimize the entire system jointly to achieve the global optimization. The semantic codec and JSC codec are trained jointly with the CE loss function to reduce the error propagation between each module.

With the Mem-DeepSC, the memory-related tasks can be performed. However, the context is transmitted via multiple time slots. If each time slot has different channel conditions, the damage to the semantic information is inevitable at the worse channel conditions, which affects the prediction accuracy. Therefore, in order to preserve the semantic information and save the communication overheads over multiple time slots, we further develop an adaptive rate transmission method.

Adaptive modulation has been developed for conventional communications [30], where the modulation order and code rate change according to SNRs. The same spirit can be used in semantic communications if there exists the relationship between the length of semantic signal and channel noise over AWGN. In this situation, we can achieve such adaptive operation by masking some elements, i.e., masking less at low SNR regimes to ensure the reliability of performing tasks and masking more elements at high SNR regimes to achieve a higher transmission rate.

How many semantic elements should be transmitted? The existing works [13, 9] employ neural networks to learn how to determine the number of transmitted semantic elements dynamically, which lacks of interpretability. Therefore, we provide a theoretical analysis of the relationship between the length of semantic signal and the channel noise to guide us to determine the number of semantic elements at certain SNR.

The key is to find the relationship between the noise level and the number of elements that can be transmitted correctly. Firstly, we model ${\bm{x}}^{c}(k)$ into

where ${\bm{r}}^{c}(k)$ is the semantic information selected from the latent semantic codewords, ${\bm{n}}_{\tt model}\sim{\cal CN}\left(0,\sigma^{2}_{m}{\bf I}\right)$ is the model noise. We generally initialize the model weights with Gaussian distribution and apply the batch normalization/layer normalization to normalize the outputs following ${\cal N}(0,1)$ [31]. Therefore, we model the model noise with Gaussian distribution. In deep learning, the model noise is caused by the unstable gradients descending, the training data noise, and so on. The model noise can be alleviated by the larger dataset, the refined optimizer, and the re-designed loss function but cannot be removed.

Assume the length of ${\bm{x}}^{c}(k)$ is $L$. By applying the packing sphere theory [32], ${\bm{x}}^{c}(k)$ can be mapped to the $L$-dimension sphere space as shown in Fig. 3(a). In the Fig. 3(a), the smaller sphere represents the noise sphere with radius ${\sqrt{L}\sigma_{m}}$ and the larger sphere is the signal sphere with radius ${\sqrt{L(\mu^{2}_{\max}+\sigma_{m}^{2})}}$, where $\mu_{\max}$ is the maximum value in the latent semantic codewords. The reason that noise spheres spread the signal sphere is that the latent semantic codewords have different constellation points. Communication is reliable as long as the noise spheres do not overlap. Therefore, there exists a minimum length of $L$ to prevent the overlap from the model noise. In other words, the number of semantic codewords that can be packed with non-overlapping noise sphere over the model noise is

After transmitting ${\bm{x}}^{c}(k)$ over the AWGN channels, the received signals can be represented by submitting (11) into (2),

where ${\bm{n}}$ in (2) is re-denoted to ${\bm{n}}_{\text{channel}}$. The ${\bm{y}}^{c}(k)$ can also be mapped to the $L$-dimension sphere space shown in Fig. 3(b). Because of the channel noise, the radius of noise sphere increases from ${\sqrt{L(\sigma_{m}^{2})}}$ to ${\sqrt{L(\sigma_{n}^{2}+\sigma_{m}^{2})}}$, which makes the noise spheres overlap.

In order to eliminate the overlapping, one way is to increase the length of ${\bm{x}}^{c}(k)$ from $L$ to $L_{1}$ ($L_{1}>L$) to enlarge the volume of the signal sphere so that the enlarged noise spheres do not overlap. Then, the number of semantic codewords that can be packed with non-overlapping noise sphere over the model noise and the channel noise is

The semantic codewords only describe the semantic information of the source and are irrelevant to the channel noise, which means that the numbers of semantic codewords in (12) and (14) are the same. Therefore, the relationship between $L$ and $L_{1}$ can be derived as shown in proposition 2.

With proposition 2, the masked ratio to different SNRs can be computed theoretically. With (15), the asymptotic analysis can be derived into four cases listed below.

• •

  Case 1: When $\sigma^{2}_{n}\to 0$, then $L_{1}\to L$. The number of transmitted symbols will converge to minimum $L$. In this case, the semantic communication system can be viewed as the compressor and decompressor.

• •

  Case 2: When $\sigma^{2}_{n}\to\infty$, then $L_{1}\to\infty$. The number of transmitted symbols will lead to infinity. In this case, the semantic communication system experiences an outage.

• •

  Case 3: When $\sigma^{2}_{m}\to 0$, then $L\to 0$. $L_{1}$ only depends on the channel noise and can be computed by

  $${\color[rgb]{0,0,0}L_{1}=\frac{2\log\left(N\right)}{{\log\left({1+\frac{{\mu_{\max}^{2}}}{{\sigma_{n}^{2}}}}\right)}}.}$$

  (16)

  In this situation, $L_{1}$ is computed by the traditional channel capacity and the number of semantic codewords.

• •

  Case 4: When $\sigma^{2}_{m}\to\infty$, then $L\to\infty$. The semantic communication system experiences an outage, similar to case 2.

The key differences between the relationship and traditional channel capacity can be summarized in the following,

• •

  The relationship indicates how much semantic information can be transmitted error-free while the traditional channel capacity indicates how many bits can be transmitted error-free.

• •

  The length of semantic vectors is affected by three points, 1) the number of semantic codewords, 2) the model noise, and 3) the channel noise. But the channel capacity only depends on the channel noise.

• •

  When channel noise disappears, the length of semantic vectors has the lower bound, $L$. The traditional channel capacity does not have such a lower bound.

With the relationship, it is possible to achieve dynamic transmission. The key to achieving such a dynamic transmission in semantic communication systems is to identify which elements are more important than the others and mask the unimportant ones. As shown in Fig. 4, we propose two mask methods subsequently, importance mask method and consecutive mask method.

As shown in Fig. 4(a), the importance mask method introduces the importance-aware model to identify the importance order among the elements of ${\bm{x}}^{c}(k)$, which can be expressed as

where $F\left(\cdot;{\bm{\theta}}\right)$ is the importance-aware model with learnable parameter ${\bm{\theta}}$, ${\bm{p}}^{c}(k)$ is the importance rank of ${\bm{x}}^{c}(k)$, in which the bigger value means that the corresponding element is more important.

By setting the threshold, $\kappa$, the mask, ${\bm{m}}^{c}(k)$, can be computed with the ${\bm{p}}^{c}(k)$ by

where $\kappa$ can be determined by the $L_{1}$-th element in the sorted importance rank with the descend order, and $L_{1}$ is computed by (15).

Then, the masked transmitted signal can be generated by

With $\tilde{\bm{x}}^{c}(k)$, the transmitter can send the only non-zero elements and the position information of zero elements to reduce the communication overheads.

After transmitting $\tilde{\bm{x}}^{c}(k)$ over the air, the receiver follows the same processing to perform signal detection, JSC decoding, and semantic decoding.

As shown in Fig. 4(b), the consecutive mask method masks the last consecutive elements in the ${\bm{x}}^{c}(k)$ to zero, so that the transmitter only sends the non-zero elements and the receiver pads the received signals with zeros to the same length of ${\bm{x}}^{c}(k)$. The consecutive mask method does not need to transmit the additional mask position information but to re-train the Mem-DeepSC model. Since the importance rank of the elements of ${\bm{x}}^{c}(k)$ is not consecutive, directly masking these consecutive elements may experience performance degradation. The Mem-DeepSC needs to be re-trained with the consecutive mask so that it can learn to re-organize the elements of ${\bm{x}}^{c}(k)$ following the order of descend importance.

The training of the consecutive mask method only includes one step, which is similar to the Train Whole Network in Algorithm 1 but with two additional operations, i.e., masking operation before transmitting and padding operation after signal detection. The loss function during the training is the CE loss function.

Fig. 5 compares the answer accuracies over different channels, in which the Mem-DeepSC and the UTF-8-Turbo transmit 32 symbols per sentence and 190 symbols per sentence, respectively. The proposed Mem-DeepSC with memory outperforms all the benchmarks at the answer accuracy in all SNR regimes by the margin of 0.8. Compared the Mem-DeepSC with memory and without memory, the memory module can significantly improve the answer accuracy, which validates the effectiveness of the memory module in memory-related transmission tasks. Besides, the Mem-DeepSC outperforms the separate Mem-DeepSC in low SNR regimes, which means that the three stage training algorithm can help improve the robustness to channel noise. From the AWGN channels to the Rician channels, the proposed Mem-DeepSC with memory experiences slight answer accuracy degradation in the low SNR regimes but the UTF-8-Turbo has an obvious performance loss in all SNR regimes. The inaccurate CSI deteriorates the answer accuracy for both methods, however, the proposed Mem-DeepSC can keep a similar answer accuracy in high SNR regimes, which shows the robustness of the proposed Mem-DeepSC.

Table II compares the number of transmitted symbols for different methods. Compared to the UTF-8-Turbo with the adaptive modulation and channel coding (AMC), the proposed Mem-DeepSC decreases the number of the transmitted symbols significantly with only 4%-16.8% symbols. The reason is that the Mem-DeepSC transmits the semantic information at the sentence level instead of at the letter/word level. Besides, applying the dynamic methods can further reduce the number of transmitted symbols from 32 symbols to 16 symbols per sentence as the SNR increases, especially saving an additional 50% symbols in the high SNR regimes. Then, the effectiveness of (15) is validated by the following simulation in Fig. 6.

Fig. 6 verifies the effectiveness of the proposed mask strategy. For Mem-DeepSC with no mask, we provided two cases with 16 symbols and 32 symbols per sentence, respectively. Utilizing adaptive modulation and channel coding (AMC) on UTF-8-Turbo can yield comparable answer accuracy to that of Mem-DeepSC over AWGN channels. However, this comes at the expense of a reduced transmission rate. Then comprising the no mask cases with different number of symbols per sentence, increasing the number of symbols per sentence leads to higher answer accuracy in low SNR regimes but the gain disappears as the SNR increases. This suggested that the semantic communication systems can employ more symbols in low SNR to improve the robustness and transmit fewer symbols in the high SNR regimes to improve the transmission efficiency. The proposed importance mask and consecutive mask keep the similar answer accuracy as the Mem-DeepSC with 32 symbols per sentence in all SNR regimes over the AWGN and the Rician channels.

However, the random mask experiences significant answer accuracy loss in the high SNR regimes as the number of masked elements increases because some important elements are masked randomly to reduce the answer accuracy. This validates the effectiveness of both proposed mask methods.