Robust Message Embedding via Attention Flow-Based Steganography

Traditional Steganography Image steganography hides information in an image by performing imperceptible changes on a host image. Traditional methods modify the image in spatial or transform domain [3]. Spatial-domain steganography generally leverages least-significant-bit (LSB) replacement [25], bit plane complexity segmentation (BPCS) [17, 28] and palette reordering [14, 29] to conceal information. However, this kind of scheme may raise statistical anomalies that can be detected by steganalysis techniques [9, 48]. Some methods utilize high-dimensional features [30] and distortion constraints [20] to improve steganography security and quality. Transform-based steganography can hide data in a transformed domain using discrete cosine transform (DCT) [2] and discrete wavelet transform (DWT) [34, 53]. Due to the limited ability of feature representation and transformation, traditional methods generally cannot achieve a satisfying quality.

Deep Steganography Recently, various deep learning-based image steganography schemes have been proposed and have achieved impressive performance. HiDDeN [52] adopted the autoencoder (AE) to embed binary messages. Baluja [3] first utilized an end-to-end network to hide a color image in another. Some studies [10, 31, 32, 37, 36, 49] incorporated generative adversarial network (GAN) [11] to reduce the image artifacts and defend steganalysis. More recently, the invertible neural network (INN) [6, 7, 18] has been widely used for steganography. These methods successfully hide single [5, 16, 24] or multiple [12, 43, 46] images in a carrier image. There are also some studies focusing on coverless steganography [21, 23, 26, 47] that directly transforms the secret information into a cover image. These methods mainly focus on improving the embedding capacity instead of robustness, as a result, they generally cannot survive image distortions.

Robust Steganography Robust steganography allows information decoding even if the images are interfered with by digital transmission or real-world distortions, which is meaningful for scenarios like copyright protection, secret communication, etc. VisCode [50] hides QR Codes in host images and can survive image brightness changes and slight tampering. LFM [42] is robust to light field messaging. RIIS [45] considers JPEG compression and various kinds of noise separately based on a conditional network. StegaStamp [35] and ChartStamp [10] take printing and photography into account but can only embed very little information at a cost of visual quality loss. As far as we know, existing methods cannot achieve both high-quality and high-capacity message steganography that is robust to extreme image distortions. In this paper, we aim to take the advantages of transformer-based model to address this problem.

Normalizing flow model was first proposed as a generative model by Dinh et al. [6]. With further improvement by RealNVP [7] and GLOW [18], it is also known as the invertible neural network (INN). INN can learn an invertible function using a set of affine coupling layers with shared parameters to map the original data distribution to a simple distribution (e.g., Gaussian distribution). Chen et al. [4] proposed an unbiased estimation for normalizing flow model. i-RevNet [15] utilizes an explicit inversion to improve the invertible architecture.

Recently, normalizing flow has been applied to various downstream tasks in computer vision, such as image [44] and video [54] super-resolution, image-to-image translation [38], etc. Especially, in the field of steganography, normalizing flow-based methods [5, 24, 46] have shown promising performance. HiNet [16] introduces the discrete wavelet transform (DWT) to guide channel squeezing and improve the steganography quality. DeepMIH [12] hides single or multiple images with a saliency detection module. Mou et al. [27] incorporated a key-controllable network design to implement secure video steganography. Xu et al. [45] simulated distortions during model training to improve the robustness and security of their method. Although previous studies have leveraged various methods to improve the network architecture for better performance, they cannot attend to both image quality and steganography robustness simultaneously.

Given a secret message $T_{s}$, we first encode it into a QR Code image $I_{q}$. The concealing procedure aims to embed $I_{q}$ into a host image $I_{h}$ and derive a stego image $I_{s}$ that is perceptually similar to $I_{h}$. Then, $I_{s}$ can suffer from various real-world image distortions, resulting in a distorted image $I_{s}^{\prime}$. After that, the revealing procedure aims to restore a QR Code $\hat{I}_{q}$ from $I_{s}^{\prime}$ that can be successfully recognized to obtain the original message.

To achieve the aforementioned targets, we first leverage a QR Code transition scheme (Sec. 3.3) to transform the original QR Code according to the host image, reducing the artifacts it causes in the subsequent steganography process. Then, we use an invertible token fusion (ITF) module (Sec. 3.4) to improve the stego image quality. After that, we propose an AttnFlow model (Sec. 3.5) to perform message embedding. To make our method robust to real-world distortions, we incorporate a distortion simulation module during the training stage, which will be described in detail in Sec. 3.6. Fig. 2 demonstrates an overview of the pipeline of our RMSteg.

Normalizing Flow [6, 7, 18], also called the invertible neural network (INN), is proposed to model a bijective projection from a complex distribution (e.g., images) to a tractable distribution (e.g. Gaussian distribution and Dirac distribution). This kind of model generally comprises several invertible affine coupling blocks (ACBs). The most basic ACB architecture is proposed by NICE [6], in which the input $u^{i}$ of the $i$^th ACB is split into two parts, $u^{i}_{1}$ and $u^{i}_{2}$, whose corresponding outputs are $u^{i+1}_{1}$ and $u^{i+1}_{2}$, respectively. For the forward process, the following transformation is performed:

$$u^{i+1}_{1}=u^{i}_{1}+\sigma(u^{i}_{2}),~{}~{}u^{i+1}_{2}=u^{i}_{2}+\delta(u^{i+1}_{1}),$$

(1)

where $\sigma(\cdot)$ and $\delta(\cdot)$ are arbitrary functions. Obviously, the backward process can be formulated as:

$$u^{i}_{2}=u^{i+1}_{2}-\delta(u^{i+1}_{1}),~{}~{}u^{i}_{1}=u^{i+1}_{1}-\sigma(u^{i}_{2}).$$

(2)

In the normalizing flow architecture, $\delta(\cdot)$ and $\sigma(\cdot)$ in Equation 1 and Equation 2 can be implemented by neural network modules with shared parameters and inverse calculation manner. By stacking multiple ACBs, the network can learn an invertible transformation between two distributions. Since this scheme is inherently suitable for steganography, many studies have utilized it for data hiding and proposed various improvements. In this paper, we further extend the ability of normalizing flow and propose a new network architecture for our robust message embedding task.

For the message embedding task in this paper, the hidden QR code needs to be restored with enough accuracy to be identified by common devices like cell phones, webcams, etc. To balance the trade-off between the stego image quality and decoding accuracy, VisCode [50] obtains a visual saliency map to guide the QR Code embedding while ChartStamp [10] utilizes the semantic segmentation result as the training loss guidance. Although this kind of rule-based strategy can improve the visual quality of the stego image, it does not consider the inherent relationship between QR Codes and the host image.

In our method, we adopt a more direct approach, which is modifying the QR Code image according to the host image (shown in Figure 2 (a)). We call it invertible QR Code transition (IQRT). The key idea of IQRT is that, the QR Code used for steganography is not necessarily to be black-and-white to keep its information. Thus, a learnable transformation can be applied to the QR Code for a better steganography quality as long as the transformed code is still identifiable. Formally, given a host image $I_{h}$ and a QR Code $I_{q}$ with the same size, we use an off-the-shelf INN architecture¹¹1We only use 2 invertible blocks instead of 16 in the original paper [24]. proposed by ISN [24] to learn an invertible function $f(\cdot)$ that derives the transformed QR Code $I_{q}^{*}$ by $I_{q}^{*}=f(I_{q},I_{h})$. In the reverse process, the restored QR Code $\hat{I_{q}}$ can be obtained by $\hat{I_{q}}=f^{-1}(I_{q}^{*},I_{s}^{\prime})$, where $f^{-1}(\cdot)$ is the inverse function of $f(\cdot)$ defined by normalizing flow and $I_{s}^{\prime}$ is the distorted stego image. Here $I_{s}^{\prime}$ is used instead of $I_{h}$ since the latter is unknown in the decoding procedure.

During network training, the transition network is jointly trained with the subsequent steganography network. We employ the same constraint as ArtCoder [33] to the transformed QR Code to ensure that it is still identifiable. Specifically, a Gaussian convolution kernel is applied to each code module to simulate the QR Code scanning procedure. For more details²²2A detailed explanation is also provided in the appendix., we suggest referring to the original paper [33]. We do not use extra constraint to the transition network so that it can learn the best transition strategy according to the overall optimization targets. Figure 3 shows some transition results, it can be observed that the transformed QR Codes have obviously lower brightness. However, with the aforementioned constraint, the transformed QR Codes are still identifiable, guaranteeing almost no information loss.

With the transformed QR Code, we first use a ViT [8] to obtain a tokenized representation $T_{q}\in\mathbb{R}^{N\times D}$, in which $N$ is the number of tokens and $D$ represents the token dimensionality. Inspired by the invertible $1\times 1$ convolution proposed by GLOW [18], before feeding $T_{q}$ to the subsequent steganography network, we put forth an invertible token fusion (ITF) module (as shown in Figure 2 (b)) to transform the QR Code tokens for a better steganography quality.

Formally, we use a learnable matrix $\mathcal{M}\in\mathbb{R}^{N\times N}$, which is initialized as an orthogonal matrix using Cholesky decomposition [19], as a transform matrix for $T_{q}$. In the steganography process, $T_{q}$ is transformed by performing a matrix multiplication: $T_{q}^{\prime}=\mathcal{M}\cdot T_{q}$. Obviously, in the decoding procedure, the restored tokens $\hat{T_{q}}$ can be obtained by: $\hat{T}_{q}=\mathcal{M}^{-1}\cdot\hat{T}_{q}^{\prime}$, where $\mathcal{M}^{-1}$ is the inverse matrix.

Different from GLOW [18] that utilizes the invertible convolution to learn the channel-wise fusion strategy, our ITF module learns a patch-wise transformation that enables inner-channel feature interaction. Our experiments also prove that ITF can efficiently and effectively improve the steganography quality by simply introducing the aforementioned learnable matrix.

Previous steganography studies based on normalizing flow generally adopt a convolutional neural network (CNN) based backbone, mostly DenseNet [40], to construct the affine coupling blocks (ACBs). This kind of design only considers the channel-wise feature fusion and can lead to perceptible artifacts in stego images, especially in the robust steganography task. Motivated by the impressive performance achieved by the transformer-based [8, 39] vision models recently, we propose a model called AttnFlow that introduces attention mechanism to normalizing flow to implement robust steganography.

As shown in Figure 2 (c), similar to ordinary normalizing flow, AttnFlow contains several attention affine coupling blocks (AACBs) for invertible function learning. Assume that the input of the $i$^th AACB is split into $T_{h}^{(i-1)}$ and $T_{q}^{(i-1)}$, corresponding to the host image tokens and QR Code tokens, respectively. Specifically, $T_{h}^{(0)}$ is the tokenized host image obtained with a basic ViT [8] and $T_{q}^{(0)}$ represents the QR Code image tokens output by the ITF module. For the $i$^th AACB, we perform the following affine transformation:

$$\begin{split}T_{h}^{(i)}&=T_{h}^{(i-1)}+\phi(T_{q}^{(i-1)})+\mathcal{C}(T_{q}^{(i-1)},T_{h}^{(0)})\times\alpha_{i},\\ T_{q}^{(i)}&=\eta(T_{h}^{(i)})+T_{q}^{(i-1)}\odot exp(\rho(T_{h}^{(i)})),\end{split}$$

(3)

in which $\phi(\cdot)$, $\eta(\cdot)$, $\rho(\cdot)$ are self-attention blocks [39] followed by a feedforward multilayer perceptron (MLP), $\mathcal{C}(q,kv)$ represents the cross-attention block [39], $exp(\cdot)$ is the exponential function, $\odot$ indicates the Hadamard product and $\alpha_{i}$ is a dependent trainable coefficient for each AACB. We calculate the attention value with:

$$Attn(Q,K,V)=M\cdot V,~{}~{}M=Softmax(\frac{QK^{T}}{\sqrt{d}}),$$

(4)

where $Q$, $K$, $V$ are derived from the learned projections and $d$ is the dimension of the projected tokens. As described in Equation 3, in addition to the self-attention value, we also calculate the cross-attention value of the initial host image tokens $T_{h}^{(0)}$ upon the QR code tokens $T_{q}^{(i-1)}$ for each AACB. Then, we add these values to $T_{h}^{(i-1)}$ to help AACBs gradually integrate the information from the QR code into the image. For the QR Code tokens, we choose to adopt a generally incorporated [5, 16, 24, 46] affine transformation and replace the original convolutional blocks with $\eta(\cdot)$ and $\rho(\cdot)$. After $n$ AACBs, $T_{h}^{(n)}$ further goes through a detokenizer³³3The detailed architecture of the tokenizers and detokenizers will be described in the appendix., resulting in the final stego image. Although some methods further map $T_{h}^{(n)}$ and $T_{q}^{(n)}$ as a conditional distribution for better performance, here we choose to adopt the same assumption as HiNet [16], which is simply positing that $T_{q}^{(n)}$ obeys a Gaussian distribution.

In the revealing process, we aim to restore the original QR Code from a distorted stego image $I_{s}^{\prime}$, we first tokenize it and derive $\hat{T}_{h}^{(n)}$. Then, we obtain $\hat{T}_{q}^{(n)}$ by sampling from a standard Gaussian distribution. After that, we perform the inverse AACB transformation by going through them with inverse calculation manner:

$$\begin{split}\hat{T}_{q}^{(i-1)}&=(\hat{T}_{q}^{(i)}-\eta(\hat{T}_{h}^{(i)}))\odot exp(-\rho(\hat{T}_{h}^{(i)})),\\ \hat{T}_{h}^{(i-1)}&=\hat{T}_{h}^{(i)}-\phi(\hat{T}_{q}^{(i-1)})-\mathcal{C}(\hat{T}_{q}^{(i-1)},\hat{T}_{h}^{(0)})\times\alpha_{i},\end{split}$$

(5)

in which $\hat{T}_{h}^{(0)}$ is obtained by tokenizing $I_{s}^{\prime}$ since $I_{h}$ is unknown in the decoding process. Then, $\hat{T}_{q}^{(0)}$ is detokenized and fed into the reversed QR Code transition (introduced in Sec. 3.3) with $I_{s}^{\prime}$ to get the final decoded QR Code image.

Distortion Simulation Module We use a module to simulate the distortions that the stego images may undergo during printing and photography. In this paper, we choose to use the same simulation module proposed by StegaStamp [35] that considers color shifting, blurring, noising, etc. We mainly modify the standard deviation of Gaussian noise from 0.02 to 0.07 and increase the JPEG compression quality from 25 to 60 for our task. During training, we perform random distortion combinations on stego images to simulate real-world image disturbances.

Loss Function The aforementioned three networks (IQRT, ITF and AttnFlow) are trained jointly. We use the following loss functions to guide the training process:

$$\mathcal{L}_{steg}^{L1}=\left\|I_{h}-I_{s}\right\|_{1},$$

(6)

$$\mathcal{L}_{steg}^{ssim}=ssim(I_{h},I_{s}),$$

(7)

$$\mathcal{L}_{steg}^{lpips}=lpips(I_{h},I_{s}),$$

(8)

$$\mathcal{L}_{qr}=\|I_{q}-\hat{I}_{q}\|_{1},$$

(9)

in which $ssim(\cdot)$ represents the structural similarity index [41] and $lpips(\cdot)$ indicates the perception loss [51]. Besides, as introduced in Sec. 3.3, an additional QR Code transition loss $\mathcal{L}_{t}$ is incorporated. The overall loss function is the weighted sum of the above functions:

$$\mathcal{L}_{total}=\alpha\mathcal{L}_{steg}^{L1}+\beta\mathcal{L}_{steg}^{ssim}+\gamma\mathcal{L}_{steg}^{lpips}+\delta\mathcal{L}_{qr}+\epsilon\mathcal{L}_{t},$$

(10)

where $\alpha$, $\beta$, $\gamma$, $\delta$, $\epsilon$ are weight coefficients.

Datasets Our training and testing datasets of host images are the train2017 (118K) and test2017 (41K) datasets of COCO [22], respectively. For QR Code images, we manually construct the training (50K) and testing (41K) datasets with random encoded messages. We generate the QR Code images by adopting the scheme of QR Code version 5 [1] with highest error correction (ECC) level of ‘H’. We incorporate this code version for most of our experiments except the evaluation in Sec. 4.3. The image used for training and testing is 224 $\times$ 224 and the patch size of ViT [8] is 16.

Metrics Our experiments focus on two aspects: stego image quality and decoding accuracy. For stego image quality, we use the peak signal-to-noise ratio (PSNR), SSIM [41] and LPIPS [51] to measure the difference between host images and stego images. For decoding accuracy, we adopt the text recovery accuracy (TRA) [46, 50], which is the ratio of the successfully decoded QR Codes. In addition, we calculate the error module rate (EMR), which represents the error rate (in percentage) of the modules in the QR Code.

Baselines We compare our method with some state-of-the-art methods⁴⁴4Since RIIS [45] has not released its source code or pre-trained model, we cannot compare with it., including ISN [24], HiNet [16] and StegaStamp [35]. Since these methods are not designed specifically for our task, we train these models on our datasets for a fair comparison. Moreover, for ISN and HiNet, since they are not robust steganography methods, we incorporate the distortion simulation module when training them. The re-trained models of these two methods are illustrated as ISN^$\dagger$ and HiNet^$\dagger$, respectively. For StegaStamp, we also additionally train it by using the same distortion level as our method, represented as StegaStamp^$\dagger$.

Steganography quality indicates both stego image quality and decoding accuracy. To compare the robustness of different methods against image distortions, we first consider several manually created situations⁵⁵5The experiments under more situations are presented in the appendix.: Gaussian noise, JPEG compression and random noise combinations. We generate random noise combinations (represented by Mixed) with the distortion simulation module introduced in Sec. 3.6. We then consider the printing and photography case to validate the methods’ robustness against real-world distortions since it is one of the most extremely severe distortion situations and contains mixed disturbance factors. We randomly select 100 host images and embed random message in them. We then use a inkjet printer to print the encoded images out and take photos with a cell phone. To eliminate the potential errors caused by factors like print quality, we repeat the experiment for 5 times and choose the best result.

Table 1 shows the experiment results, Figure 4 and Figure 5 demonstrate some qualitative results. It can be observed that our method can achieve higher stego image quality, especially for LPIPS that represents the perceptual similarity. StegaStamp incorporates adversarial training [11] to make the generated stego image more realistic and it does work when facing low-level distortions. However, with the distortion used during training increases, StegaStamp^$\dagger$ fails to preserve a sound visual quality and instead lead to hue shifting and artifacts. In addition, adversarial training can sometimes bring severe artifacts in some regions, as shown in the 3rd and 4th row of Figure 4. For HiNet and ISN that both leverage normalizing flow, due to the lack of inner-channel interaction by using CNN-based affine block, the stego images they derive have obvious QR Code-like artifacts, making the existence of secret message easy to detect. In terms of decoding accuracy, although HiNet^$\dagger$ outperforms our method in some cases, we achieve the best performance in the mixed noise and printing tests, which are more close to the real-world application scenarios.

Figure 7 shows the decoding accuracy under more levels of distortions, which are Gaussian Noise whose standard deviation ranges from 0.02 to 0.2 and JPEG compression with quality ranging from 10 to 90. It can be observed that our method demonstrates a stable and good performance under these situations.

In practical application scenarios, the shooting condition may vary from time to time and a good message embedding method should keep its robustness in most cases. As a result, we further measure the decoding accuracy under different shooting situations. We mainly consider the shooting distance and angle (the offset relative to vertical shooting). Given the default shooting distance and angle of this paper as 11cm and $0^{\circ}$, we gradually increase these two values and the test results are shown in Table 2. It can be found that, with the shooting distance and angle grow, the EMRs exhibit a significant increase for all methods. However, for TRA that directly reflects the identifiability of QR Codes, our method maintains a fair performance. The comparison shown in Figure 6 also indicates that RMSteg can achieve high decoding accuracy under different shooting situations.

To validate the generality of our method, we test it under different embedding capacity, i.e., using QR Codes with different versions for training. We demonstrate the model performance on code versions from v5 to v8 in Table 3. Although the steganography quality is getting worse with the embedding capacity increases, the artifacts in the stego images are still not perceptible, especially when the image is printed out, as shown in Figure 8. In addition, RMSteg keeps a TRA of more than 0.8 even for code v8 whose embedding capacity is two more times higher than v5. Compared with StegaStamp [35] that is also designed for robust message embedding, its PSNR is lower than 25 when encoding 200 bits in a $400\times 400$ image according to the original paper. In contrast, our method is able to keep a PSNR of around 30 when encoding more than 600 bits in a $224\times 224$ image. Thus, RMSteg can achieve a higher steganography quality and meanwhile the embedding capacity is much larger.

We conduct an ablation study to validate the effectiveness of the invertible QR Code transition (IQRT), the invertible token fusion (ITF) module and the AttnFlow model. The result is shown in Table 4.

IQRT The model without IQRT performs slightly better than the full model in decoding accuracy. This is because IQRT may sometimes cause information loss, e.g., some code module could be wrongly transformed during this procedure, although the QR Code is still identifiable. On the other hand, IQRT can largely improve stego image quality.

ITF Module As discussed in Sec. 3.4, the ITF module can learn a transformation for image tokens and thus leading to better stego image quality. We also find that the ITF module can help derive a better distribution of artifacts brought by message embedding. As shown in Figure 9, the stego image generated using ITF has much less distortion in homogenous regions (i.e., the sky), achieving a better visual quality.

Cross Attention in AACB We train our model by removing the cross attention blocks defined in Equation 3 and the result shows that this design can improve the overall model performance.

Tokenized Representation We replace the AttnFlow model with ISN and HiNet (two CNN-based normalizing flow model), respectively, to validate the effectiveness of introducing tokenized image representation (the IQRT module is retained). The result shows that, compared with CNN-based scheme, incorporating tokenized image representation makes normalizing flow more competent for robust steganography task.

AACB Number It shows that, increasing the number of AACBs can obtain a better model performance, which is consistent with the fundamental of normalizing flow.