Towards Robust Image-in-Audio Deep Steganography

Given a time domain sequence $x[n]$, its discrete short-time transform operation is given by Equation 2:

which is a bivariate function representing the energy of the $F_{k}$-th frequency component (equispaced frequency samples $F_{k}=k/N$) in the $m$-th frame. The time index is represented by $n$; $h$ is the low pass window function, and $r$ represents the hop size of the short-time transform. The function $\mathcal{T}[n,F_{k}]$ is the basis function of the transformation.

The STDCT, used by PixInWav, is a real transform using the type-2 DCT basis function as $\mathcal{T}[n,F_{k}]$, which produces a single 2D spectrogram from a 1D audio waveform. On the other hand, the STFT is a complex transform using the complex exponential function as its basis function $\mathcal{T}[n,F_{k}]=e^{-i2\pi F_{k}n}$, which results in a complex signal that can be split into two 2D signals: the magnitude and the phase. We propose using the STFT instead of STDCT, where this duality of the cover signal allows for more possibilities in how the secret signal can be embedded onto the cover signal. Both magnitude and phase, can be used as stego signals in the same manner that the single spectrogram from the STDCT has been used. In this work we consider using any of the two signals as a single stego signal, or using both of them at the same time.

STFT magnitude as a single stego signal. One can approach embedding the information in the STFT magnitude in the same way as using a single STDCT spectrogram, with the main difference being that the imaginary part of the cover signal remains unmodified. Since we are not directly distorting the phase component, the alignment performed by DTW can be deemed redundant. We propose changing the soft DTW used in [19] by a simpler $L_{1}$ distance between the cover and stego waveforms (Equation 3):

STFT phase as a single stego signal. Since the phase has the same spatial dimensions as the magnitude signal, the same methods can be applied only on the phase component. In this case, only the imaginary part is modified.

STFT magnitude and phase as stego signals. A more advanced setup has been developed in which both the STFT magnitude and phase can jointly serve as a stego signal. To handle multiple stego components, the architecture requires an adaptation: the two stego components should be treated separately due to their very different structure. The proposed architecture uses different encoders and decoders for each stego component. The two revealed images are fed into a third network that processes them to obtain a single image as output. Out of the multiple solutions tried, a simple trained weighted average worked the best. The loss function has been accommodated for the possibility of using multiple containers (Equation 4):

where $M$ and $P$ now denote the magnitude and phase signals, respectively, and $M^{\prime}$ and $P^{\prime}$ correspond to their respective stego components. These components are weighted by a new hyperparameter $\theta$, that controls the trade-off between magnitude and phase distortion. Notice that the waveform $w$ is still unique.

We consider the case of encoding a $256\times 256\times 3$ RGB image (flattened to $512\times 512$ after applying the pixel-shuffle operation) onto a $1024\times 512$ spectrogram.

PixInWav [19] made use of bilinear interpolation for upsampling the encoded image to match the spectrogram shape, only to be later downsampled to its original size before decoding. This strategy, Stretch from now on, makes the encoding and decoding processes independent of the stego size. However, other options could be devised. In this section we propose alternative architectures to address this problem (Figure 3), that make a better use of the space available by encoding multiple copies of the encoded image, and improving the secret reconstruction and increasing the robustness of the steganographic method.

Replicate method. Our simplest approach uses the fact that the cover spectrogram is significantly larger than the secret image and this allows for a natural replication of the encoded image that is added onto the host signal. When decoding, the two copies are jointly forwarded through the network, split and averaged to produce the final revealed image.

Weighted Replicate method. Weighted Replicate (W-Replicate) improves the previous method by scaling each replica by a trainable weight before adding them onto the container spectrogram, and also when merging them into a single one (essentially, a trained weighted average); resulting in a total of four trainable weights that are added to the model. This change allows the model to learn in which half of the STFT spectrogram (high or low frequencies) the information can be added causing the least distortion.

Weighted & Split Replicate method. The previous two methods decode the spectrogram directly, i.e. a tensor of shape $1024\times 512$, with the two replicas side by side; this forces the network to treat both replicas equally. Weighted & Split Replicate (WS-Replicate), improves upon this issue by first splitting the container signal and decoding the two replicas separately (i.e. concatenated in a 3rd dimension, resulting in a tensor of shape $512\times 512\times 2$). The encoder structure is the same as in W-Replicate.

Multichannel method. All the previous methods rely on the pixel-shuffle operation to flatten the image into a single color channel. Multichannel, however, omits this step and has the model learn to encode the three color channels in the different replicas. Thus, the encoded image is of shape $256\times 256\times C$, being $C$ the desired number of output channels. Since eight $256\times 256$ replicas can fit into a $1024\times 512$ host signal, we set $C=8$. They are arranged in a $4\times 2$ grid. As with WS-Replicate, the decoder is fed on the replicas already split and concatenated, only that this time the output is already the final $256\times 256\times 3$ RGB image.

Our baseline system assumes a stego signal of size $1024\times 512$, which is determined by the STFT applied on the input audio waveform with a set of hyperparameters (frame length and hop size). However, these values are arbitrary and could be changed, specifically, to increase the resolution of the stego spectrogram. In this section we explore the possibilities offered by the use of a larger spectrogram, which should increase the embedding capacity of the stego signal (and the robustness of the whole system if replication is used). Increasing the frame length of the STFT results in a larger size in the frequency dimension, while reducing the hop size of overlapping windows causes the container to increase in the time dimension. We applied both modifications to obtain a container of size $2048\times 1024$, preserving the property of the dimensions being powers of two, allowing for efficient computations. Some adaptations have been required to accommodate for the larger container size:

Stretch method needs to interpolate to a larger target size, the same as the stego spectogram.

Replicate-based methods use 8 replicas instead of 2, which are arranged in a $4\times 2$ grid. As a consequence, W-Replicate and WS-Replicate, use 8 weights instead of 2 to scale each copy individually. WS-Replicate’s decoder also needs to accept input of depth 8 instead of 2.

Multichannel’s encoder outputs 32 replicas instead of 8; the decoder also expects an input of depth 32. These are arranged in an $8\times 4$ grid.

The pixel-shuffle operation [34] used to flatten the image arranges each $1\times 1\times 3$ pixel into a $2\times 2\times 1$ grid. Thus, for every RGB pixel, we obtain a $2\times 2$ grid where we can buffer the values. PixInWav [19] padded a value of 0 into the fourth component of every grid. Contrary to this zero-padding approach, we propose padding with the luma component of the pixel in question instead, as a way to add redundancy to the signal that can be later used for error correction in the decoder side. This peculiar pixel-shuffle step then outputs, for every pixel, a $2\times 2$ grid of 4 values $[R,G,B,Y]$, where $Y$ represents the luma component of the pixel, computed from the RGB components by a standardized transformation to the YCbCr color space [21]. On the decoder side, for each $[R,G,B,Y]$ pixel, the YCbCr representation is computed from the RGB values. The newly computed $Y$ value is then averaged with the received $Y$, and the whole pixel is transformed back to the RGB color space to yield the final image.

We have used a subset of 10,000 color images from the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012), sampling 10 images per ImageNet class, [15]. Every image has been cropped and scaled to $256\times 256\times 3$, normalized, and paired with the STFT transformation of an audio clip (roughly 1.5s) sampled at 44,100 Hz from the FSDnoisy18k dataset [18]. The audio dataset contains a variety of different sounds, ranging across 20 different classes, among which we can find voice, music and noise.

In this section, we present the results of our ablation study to determine the impact of the proposed enhancements (Figure 2). Table 1 summarizes the results of our experiments and Figure 4 displays a selection of visual examples. We provide a numerical identifier for each of the models for reference.

STFT instead of STDCT. We assessed the impact of using the magnitude of the STFT as a stego signal instead of the STDCT spectrogram as in [19]. Comparing STDCT models #1–#5 against STFT models #6–#10 we can see that STDCT models struggle to find a balance point with good reconstruction of both, image and audio, while STFT models show a good performance in both.

Modification of the loss function. As a result of our choice of STFT over STDCT, we compared the results of computing the waveform loss using the $L_{1}$ distance instead of the soft DTW discrepancy. In Table 1 we can compare DTW models #6, #18, #22, #25 and #30, with $L_{1}$ models #7, #17, #21, #24 and #29. Using the $L_{1}$ distance, results were slightly better in most cases. Note that the soft DTW loss proved superior when using the STDCT, as seen when comparing models #4 and #5, explained by the reasoning in Section 4.1.

Type of STFT stego signal. Next, we compared the performance of the steganographic operation based on the kind of the stego signal: just magnitude, just phase, or a combining both. We find that using the phase as the sole container is clearly inferior to using the magnitude (compare models #9 and #10 in Table 1), as there is a very significant drop in both image and audio reconstruction quality. Our reasoning for these results is twofold. Firstly, the phase is a much noisier signal in nature, which makes the task of hiding information more difficult. Secondly, minor modifications to the phase component result in a more perceptible distortion in the reconstructed audio, thus rendering the task of concealing the secret signal more challenging. Finally, we compared using both the magnitude and phase as stego signals simultaneously (model #11). The results from Table 1 show that while using both stego signals does substantially increase image quality, there is a significant drop in audio quality, possibly as a consequence of additionally distorting the phase. In conclusion, our study suggests that it is not worth using the phase as a stego signal, since it does not improve the metrics obtained with the baseline model that only uses the magnitude. Therefore, there is no justification for the added overhead in the model.

Comparison of embedding methods. A comparison of different embedding methods has been conducted, and the results are presented in Table 1 accross models #11–#15 (stego signal of size $1024\times 512$) and models #16, #20, #23, #28, #33 (stego signal of size $2048\times 1024$). The Multichannel method exhibits a considerable enhancement in image quality, albeit at the cost of a significant decrease in audio metrics, thereby rendering it less practical for most real-world applications. Conversely, all replicate-based embedding methods outperform the baseline Stretch approach. Qualitative assessments depicted in Figure 3 demonstrate that WS-Replicate can generate a superior reconstruction of the original image, as it is the only method capable of preserving the authentic color.

Buffering the luma component in the pixel shuffle operation. Experiments shown by pairs of models #16 and #17, #20 and #21, #23 and #24, #28 and #29 show that this addition does improve the quality of the revealed images while maintaining a comparable audio quality with regard to the baseline model.

Higher resolution stego signal. The values from Table 1 show a very significant improvement when using a larger resolution of the stego signal (compare models #6–#15 against #16–#33), both in image and audio quality, as it is expected from having more capacity for carrying information. This increase in performance comes at the cost of increased memory usage and longer training times. Note that, for training purposes, the audio transform can be precomputed for every audio; however, the inverse transform is still needed to compute the waveform loss.

To evaluate the effect of different embedding methods on the robustness of the steganographic method, we attempted to decode smaller temporal segments of the stego signal. In our experiment, we selectively zero out the spectral content at different time frames of the stego spectrograms, simulating a scenario where some data is lost during transmission [40, 20], either in large contiguous chunks or at random positions.

The qualitative results can be appreciated in Figure 5. They show that methods that use replication are more robust than the baseline Stretch approach, allowing to recover most of the image even if a large part of the stego signal is lost.

The results from the previous sections show that some setups obtain better performance than others. However, in some cases this comes at the cost of increased computational load, during both training and inference. The trade-off between these two factors should take into account the available resources and future use of the model. Table 2 presents our results of such analysis. An increase in the number of parameters implies a higher memory usage and longer execution times, and an increase in Giga Multiply–accumulate operations (GMAC) operations generally indicates longer execution times and energy consumption. Among models with similar performance, lower values in both metrics should be preferred.

Cost of using the STFT instead of STDCT. For both transforms there exist efficient algorithms with equal asymptotic complexity [11, 8]. We thus consider the two options to be equal in this regard. However, there is an additional cost if we use the magnitude and phase together as stego sigals – this scenario doubles the number of parameters and MAC operations, since two separate encoder and decoder networks are used (plus a small coupling network).

Choice over $\bm{L_{1}}$ and DTW. The usage of the $L_{1}$ loss instead of dynamic time warping cannot be directly assessed, since this is only used during the training process and outside the model. It should be noted, however, that the $L_{1}$ loss is generally much more efficient to compute ($\mathcal{O}(n)$ time) than (soft-)DTW ($\mathcal{O}(n^{2})$ time) [13].

Cost of embedding methods. The different embedding methods can also be compared in terms of computational load (Table 2). Replicate does not add any additional parameters with respect to the baseline Stretch method, the only difference being that the information is duplicated and concatenated instead of being upsampled. W-Replicate adds four additional parameters that scale each of the two replicas (which is done at both, the encoder and decoder end). The effect on the load is negligible. WS-Replicate and Multichannel methods use deeper convolution kernels over smaller resolution tensors, thus increasing the amount of parameters while decreasing the total number of operations. This is especially noticeable in Multichannel.

Using higher resolution of stego signal. When using a larger stego spectrogram, the amount of parameters remains the same, except for WS-Replicate and Multichannel, that use even deeper kernels to process a larger number of replicas. However, the number of floating-point operations triples.

Cost of buffering the luma component. The usage of the luma channel in the pixel-shuffle operation only entails a color space change (done through a single matrix multiplication) and averaging the two luma values. These operations do not add any extra trainable parameters, and the computational cost is negligible.