Autoregressive Linguistic Steganography Based on BERT and Consistency Coding

As shown in Fig. 1, the proposed autoregressive LS method involves three participants: the data hider Alice, the adversary (attacker) Wendy, and the data receiver Bob. The goal of Alice is to hide a secret message (in the form of a binary stream) into a cover text. The resulting stego text carrying the secret message will be sent to Bob via an insecure channel such as the Internet, which is being monitored by the adversary Wendy whose purpose is to determine whether a text being transferred contains secret information or not.

Specifically, according to the secret key and the secret message to be embedded, Alice first determines a set of masked positions in the cover text. For each masked position to be processed, a word will be determined according to the masked LM, the present temporary text (which may contain processed and unprocessed masked positions) and the consistency coding technique so that the word will substitute the masked position and meanwhile match the secret bits to be embedded. When all masked positions are processed, a stego text can be generated and will be sent to Bob, who performs the operation similar to Alice on the stego text so that the entire secret message can be extracted from the stego text. In order to ensure that the data extraction procedure can be successfully executed, Alice and Bob should share the necessary side information in advance including the secret key, the threshold controlling the selection of candidate words, the masked LM as well as the consistency coding strategy. In the following, we provide more details.

The goal of the masking strategy is to replace some words (tokens) in the cover text with the special token “[MASK]”. These masked positions will be filled with tokens generated by the masked LM based on the context. The new tokens not only fit into the context, but also carry secret information. It is free for us to design the masking strategy. Though the masking strategy can be carefully crafted, it is not the main contribution of this paper. For fair comparison in experiments, we follow the simple but effective masking strategy in [27]. In brief summary, a masking interval $f$ (which is an integer and can be considered as a secret key) is used to control the total number of masked positions. A higher $f$ means less tokens are masked, which reduces the embedding capacity but increases the difficulty of detection.

The masked strategy has been previously introduced along with BERT [28] as an efficient pretraining strategy for Transformer based nets [29]. By using the masked strategy, BERT overcomes the limitations of unidirectional training and realizes the common dependence on context for prediction. Generally speaking, fine-tuning the downstream tasks with the pre-trained BERT according to their specific tasks such as sentence classification, sequence tagging and question answering, can greatly reduce the expense of designing their own architecture. However, fine-tuning is not required in this paper. Different from recurrent neural networks (RNNs) [30] processing texts by a word-wise manner and generative pre-trained transformer (GPT) [31] obscuring future markers, the masked LM provides superior prediction performance due to its bidirectionality. Therefore, we use the pretrained BERT as the masked LM.

Let $\mathcal{M}$ and $\mathcal{I}=\{i_{1},i_{2},...,i_{s}\}$ be the masked LM and the index-set for the masked positions determined with the secret key. For each $i\in\mathcal{I}$, the corresponding word $x_{i}$ in the cover text $\textbf{x}=\{x_{1},x_{2},...,x_{n}\}$ will be replaced with “[MASK]”. To express it mathematically, for any $\mathcal{S}\subset\mathcal{I}$, we define $\textbf{x}_{\mathcal{S}}$ as:

where for $1\leq i\leq n$, we have

Obviously, we have $\textbf{x}_{\emptyset}=\textbf{x}$ and $\textbf{x}_{\mathcal{I}}$ is the text after replacing all the words in the masked positions with “[MASK]”. To embed secret information in the masked positions, we orderly process the masked positions. For each masked position, $\mathcal{M}$ will output a prediction probability for each word in the vocabulary according to the temporary text generated previously. A higher probability implies that the corresponding word is more suitable to replace the “[MASK]” to fit into the context. Next, we determine the temporary text for each masked position.

Without the loss of generalization, let $i_{1}<i_{2}<...<i_{s}$. The temporary text for the masked index $i_{j}\in\mathcal{I}$ is determined as follows. First, the temporary text for $i_{1}\in\mathcal{I}$, denoted by $\textbf{x}_{[i_{1}]}$, is $\textbf{x}_{\mathcal{I}}$. By feeding $\textbf{x}_{[i_{1}]}$ to $\mathcal{M}$, we can generate a prediction probability $p_{i}\in[0,1]$ for each $v_{i}\in V=\{v_{1},v_{2},...,v_{z}\}$ and $\sum_{i=1}^{z}p_{i}=1$. According to the secret bits to be embedded and the proposed consistency coding technique, we select a word $x_{i_{1}}^{*}\in V$ to replace the “[MASK]” in the $i_{1}$-th position in x. For $i_{2}\in I$, the only one difference between $\textbf{x}_{[i_{1}]}$ and $\textbf{x}_{[i_{2}]}$ is that in the $i_{1}$-th position, $\textbf{x}_{[i_{1}]}$ uses “[MASK]” but $\textbf{x}_{[i_{2}]}$ uses $x_{i_{1}}^{*}$. More general, the only one difference between $\textbf{x}_{[i_{j-1}]}$ and $\textbf{x}_{[i_{j}]}$ is that in the $i_{j-1}$-th position, $\textbf{x}_{[i_{j-1}]}$ uses “[MASK]” but $\textbf{x}_{[i_{j}]}$ uses $x_{i_{j-1}}^{*}$ for any $2\leq j\leq s$.

Taking Fig. 1 for explanation, the cover text is “Midshire is a wonderful little city .” and there are two masked positions. For the first masked position (from left to right), the temporary text to be fed into the masked LM is “Midshire is a [MASK] little [MASK] .”. After the first masked position was replaced with the stego word “nice”, the temporary text to be fed into the masked LM for the second masked position is “Midshire is a nice little [MASK] .”. It is seen that the temporary text for the present masked position depends on the generation of the previous temporary text. We deem it an autoregressive method.

A most important problem is how to build the mapping relationship between words and secret bits. To deal with this problem, the authors in [27] use a threshold $0\leq t_{p}\leq 1$ to collect a list of candidate words whose prediction probabilities are higher than $t_{p}$, and then map each of the candidate words to a fixed-length binary stream. In other words, the collected candidate words have the same probability of being selected to fill in the present masked position, which does not take into account the probability distribution of the candidate words and may not fit into the context well.

As mentioned previously, it is quite desirable to select such a word that it not only matches the secret bits to be embedded but also has a prediction probability obtained from the masked LM as high as possible. Because the secret bits are evenly distributed, the probability of selecting a specific word as the output actually depends on the number of secret bits carried by the word. In other words, for any two words mapped to a binary stream with the same length, they often have the same probability of matching the secret stream to be embedded. For example, in [27], if all candidate words are mapped to a binary stream with a length of $l>0$, the probability of choosing any one among $2^{l}$ candidate words as the present output is $1/2^{l}$. It indicates that once the mapping relationship is finished, we are no longer free to choose the word. Therefore, it is necessary that the construction of mapping relationship itself has taken into account the prediction probability distribution so that for those words with a high prediction probability, the probability of choosing any one of them is high as well.

In summary, we expect to find such an information encoding strategy that the probability of choosing a word as the present output is proportional to its prediction probability obtained by the masked LM. We treat such information encoding strategy as consistency coding. Compared with the method introduced in [27], consistency coding has two significant advantages: 1) the number of candidate words can be any integer, rather than a power of two which is required in [27], and; 2) it bases on the statistical probability distribution of each word in the vocabulary, which takes into account the frequency of words and makes coding more conducive to the regularization of the text. Such a mechanism imitates the priority of native speakers in choosing words, and improves the security accordingly.

It is free for us to design the consistency coding. However, many classical entropy encoding methods in information theory can be used to realize consistency coding such as Huffman coding, arithmetic coding and Shannon-Fano coding [32]. For simplicity, we use Huffman coding to provide the off-the-shelf solution. Formally, for compactness, for each of the specific masked positions, let $\{p_{1},p_{2},...,p_{z}\}$ be the prediction probabilities for the words in the vocabulary $V=\{v_{1},v_{2},...,v_{z}\}$, where $p_{1}\geq p_{2}\geq...\geq p_{z}$ and $\sum_{i=1}^{z}p_{i}=1$. We first collect all the words whose prediction probabilities are higher than a pre-determined threshold $t_{p}$. The collected words are then regarded as the candidate words to be encoded. Then, by normalizing the prediction probabilities of the candidate words, we further apply Huffman coding to map each candidate word to a stream. For example, let $V^{\prime}=\{v_{1},v_{2},...,v_{w}\}\subset V,~{}(w\leq z)$, include the candidate words, the normalization operation is:

which enables us to thereafter directly apply Huffman coding.

Remark 1: When the entire secret message cannot be carried by a single cover text, multiple cover texts can be used. When the entire secret message can be carried by a single cover text, some masked positions in the cover text will be embedded with secret bits while the other masked positions (if any) are filled with the original words without data embedding.

Remark 2: It is known that Huffman coding does not result in the unique mapping. For example, in Fig. 1, the two words “city” and “town” are mapped to “0” and “1”, respectively. It is also possible that “city” is mapped to “1” and the other is mapped to “0”. As long as the data hider and the data receiver use the same secret key to control the Huffman coding process, they can obtain the same mapping result.

We first show some examples to evaluate the quality of the stego texts generated by the proposed method. Table I shows the experimental examples, in which we compare the proposed method with [27] due to the close correlation mentioned in Section II. The threshold $t_{p}$ defined in Section II and III is empirically set to 0.02 by default. The embedding parameter $f$ controls the number of the masked positions. The higher $f$ is, the lower the number of the masked positions is. The payload is measured by bits per word (bpw). Since the masking strategy always skips punctuation, the punctuation marks never carry data. For simplicity, the punctuation marks are excluded when to determine the total number of words in a text, e.g., in Table I, when $f=2$, the payload of the stego text generated by [27] is 13/31 = 0.42, rather than 13/32 = 0.41. In Table I, “$W(a,b)$” indicates that the word “$W$” is mapped to a binary stream $b$ and the number of the candidate words for the present masked position is $a$ during data embedding. For example, “ways(2,1)” means that the word “ways” carries only one secret bit “1” and there are 2 candidate words for the present position. “exploring (0,-)” means that there has no candidate words for choice and the original word “exploring” is used for the present position.

It can be inferred from Table I that compared with [27], the proposed method not only improves the semantic quality, but also increases the payload for all the three cases, which have verified the superiority of the proposed method. This can be explained from two aspects. On the one hand, compared with the non-autoregressive prediction strategy in [27], the proposed autoregressive prediction strategy enhances the semantic connection between contexts so that those words better fitting into contexts can be collected for data embedding. On the other hand, compared with block coding [27], the proposed consistency coding strategy is based on the prediction probability distribution of each word in the vocabulary, which enlarges the candidate set and makes the choice of stego words more conducive to maintaining the text quality.

To evaluate the performance against steganalysis, following [27], for each experiment, we first split the above-mentioned 10,000 natural texts and their stego versions to three disjoint subsets, i.e., training set (60%), validation set (10%), and testing set (30%), and then finetune the $\text{BERT}_{\text{base-cased}}$ model with the training set and the validation set. The finetuned model is used to detect whether a text to be tested is stego or not. The accuracy is evaluated on the testing set with the model of the highest validation accuracy. We compare the proposed work with four state-of-the-art methods, i.e., [27], [26], [22] and [20]. Table II shows the experimental results. In Table II, the parametric settings are described as follows. Ref. [27] and our work use the same $f=3$ and $t_{p}=0.02$ for fair comparison. The explanation for $f$ and $t_{p}$ has been described in Subsection IV-A. $\tau$ is a system parameter introduced in [26] and we use the recommended value in [26], i.e., $\tau=0.7$. The authors in [22] propose two methods called fixed-length coding and variable-length coding for information encoding. We use the variable-length coding strategy for simulation since it has the better performance. For [22] and [20], the truncation length $L$ and the block size $B$ are set to $2^{2}$ for fair comparison as well. As shown in Table II, different methods have different performance. In terms of payload, the methods in [26], [22], [20] outperform [27] and the proposed method. The reason is that the three methods are actually generation based while [27] and the proposed method are actually modification based. Generation based methods often provide a high payload since each word can be used to carry secret information. However, in Table II, the detection accuracies of the three generation based methods are high, meaning that the security is not satisfied. By comparing with [27], we can find that the proposed work not only achieves a higher payload, but also has a lower detection accuracy, indicating that the proposed work achieves a better trade-off between payload and security.

To further demonstrate the superiority of the proposed work, we visualize the statistical distribution of the stego texts and the natural (cover) texts by applying t-SNE [35]. It is observed from Fig. 2 that when the payload (i.e., BPW) decreases, more points of the two colors overlap, implying that the stego texts are more approximate to the natural ones and therefore indeed have higher security. For the different parameters given in Fig. 2, we also show the corresponding mean payload and detection accuracy for [27] and the proposed work. As shown in Table III and Table IV, the proposed work outperforms [27] for all cases, which has verified the superiority of the proposed work. We also determine the mean PPL of stego texts due to different parameters. The experimental results are plotted in Fig. 3. It can be inferred that the proposed work also generate stego texts with better quality than [27], indicating that it is more difficult for a person to distinguish between stego texts generated by the proposed work and natural texts compared with [27], which reveals that the proposed work has high-level concealment.

The proposed work achieves superior performance by using an autoregressive strategy and a consistency coding technique. In order to show that both the autoregressive strategy and the consistency coding technique have the ability to improve the security or the payload, we further conduct ablation study for evaluation. The method in [27] use the non-autoregressive strategy and block coding technique for steganography. By substituting non-autoregressive with autoregressive, we can develop a new steganographic method named as “Autoregressive only”. Similarly, by substituting block coding with consistency coding, a new steganographic method named as “Consistency coding only” can be simulated. Table V shows the experimental results. In terms of payload, “Consistency coding only” and the proposed work are close to each other, but both higher than [27] and “Autoregressive only”. It indicates that the proposed work can benefit from the consistency coding technique so as to increase the payload. In terms of steganalysis accuracy, “Autoregressive only” and the proposed work have the similar performance, but both outperform [27] and “Consistency coding only”. It means that the autoregressive strategy is more secure than the non-autoregressive strategy. By combining the autoregressive strategy and the consistency coding technique, the PPL value of the proposed work is the lowest in most cases. In summary, it can be concluded that the proposed work has achieved the best performance compared with related works.