FastStamp: Accelerating Neural Steganography and Digital Watermarking of Images on FPGAs

Digital watermarking techniques broadly seek to generate three different types of watermarks: fragile (Di Martino and Sessa, 2019), robust ,(Cox et al., 1997; Shehab et al., 2018; Zhu et al., 2018) and semi-fragile (Lin et al., 2000; Yu et al., 2017; Ho and Li, 2004). Fragile and semi-fragile watermarks are primarily used to certify the integrity and authenticity of image data. Fragile watermarks are used to achieve accurate authentication of digital media, where even a one-bit change to an image will lead it to fail the certification system. In contrast, robust watermarks aim to be recoverable under several image manipulations to allow media producers to assert ownership over their content even if the video is redistributed and modified. Semi-fragile watermarks combine the advantages of both robust and fragile watermarks and are mainly used for fuzzy authentication of digital images and identification of image tampering (Yu et al., 2017).

Prior work has proposed hand-engineered pipelines to embed semi-fragile watermark information in the spatial(Xiao and Wang, 2008) and frequency domain of images and videos. Particularly, in the frequency domain, the watermark can be embedded by modifying the coefficients produced with transformations such as the discrete cosine transform (DCT) (Preda and Vizireanu, 2015; Ho and Li, 2004) and discrete wavelet transform (DWT) (Tay and Havlicek, 2002; Li et al., 2015; Benrhouma et al., 2015). For example, during 2D DWT an image is first decomposed into various frequency channels using a Haar filter. A scaled image is used as the watermark and inserted into mid frequency wavelet channel. Taking 2D inverse DWT of the altered wavelet decomposition produces the watermark embedded image. The major drawbacks of traditional approaches lie in higher visibility of the embedded watermarks, increased distortions in generated images, and low robustness to compression techniques like JPEG transforms as discussed in Section 5.2. Specifically, since digital images shared over social media are highly compressed and undergo various lighting and color adjustments, the watermarks generated by DWT and DCT algorithms break under benign real-world transformations and compression.

More recently, CNNs have been used to provide an end-to-end solution to the watermarking problem. They replace hand-crafted hiding procedures with neural network encoding (Baluja, 2017; Hayes and Danezis, 2017; Zhu et al., 2018; Zhang et al., 2019; Wang et al., 2021; Tancik et al., 2020; Neekhara et al., 2022; Luo et al., 2020). These techniques train end-to-end encoder CNNs to embed and decode watermarks, which have resulted in lower imperceptibility and more robust recovery of the watermark data. However, these models are memory intensive and much slower as compared to DCT based algorithms. In our work, we address the performance limitations of neural watermarking systems and propose a light-weight framework for both robust and semi-fragile image watermarking suitable for hardware acceleration platforms.

There have been several efforts to accelerate neural networks on FPGAs (Zhang et al., 2015; Suda et al., 2016; Samragh et al., 2017; Hussain et al., 2021a). Equipped with the necessary hardware for basic DNN operations, FPGAs are able to achieve high parallelism and utilize the properties of neural network computation to remove unnecessary logic. Some prior efforts have been made in FPGA acceleration of convolutional autoencoder architectures (Liu et al., 2018; Zhao et al., 2018; Govorkova et al., 2022). While these works implement many sub-blocks used in neural network computation, we find that they cannot be directly used for an image watermarking framework because efficient implementations of sub-blocks like 2D upsampling and separable convolutions with skip-connections are absent. Moreover, existing neural architectures for image watermarking are not optimized for hardware-software co-design.

Prior work (Kiran et al., 2013; Hazra et al., 2018; Hajjaji et al., 2019; Khoshki et al., 2014) has made significant efforts in accelerating traditional image watermarking schemes that hide secret information in the frequency domain of images and rely on DCT and DWT based algorithms. As discussed earlier, while such algorithms are vastly popular for hardware applications due to its simplicity and low computational overhead, the resulting watermarked image is often not robust to real-world image transformations. In our work, we develop the first FPGA accelerator platform for DNN based image watermarking and steganography that enables robustness to real-world digital image processing and compression while being selectively fragile to media tampering techniques.

With the widespread development of deep learning based image and video synthesis techniques (Liu et al., 2019; Choi et al., 2020; Mirsky and Lee, 2021; Kowalski, 2018), it has become increasingly easier and faster to generate high-quality convincing fake images/videos such as Deepfakes. Such manipulated media can fuel misinformation, defame individuals and reduce trust in media. While considerable research effort has been made in designing CNN based deepfake detectors (Dolhansky et al., 2020; Afchar et al., 2018), these techniques have been shown to hold major security vulnerabilities and can be bypassed by attackers (Hussain et al., 2021b). To counter such threats, authors (Qureshi et al., 2021; Neekhara et al., 2022; Wang et al., 2021) have proposed semi-fragile watermarking as a solution to perform media authentication and distinguish deepfake media from real media by verifying a secret watermark embedded in the media. For both fragile and semi-fragile techniques, a watermark must be inserted when the image is captured, which makes these techniques dependent on both algorithmic and hardware implementation. If watermark information is embedded separately in images and videos after it is captured by a device, this method may fail in situations where tampering is carried out before inserting the signature or watermark. While the proposed solution to media authentication is to add a verifiable digital signature or watermark to an image/video using neural networks, prior works (Neekhara et al., 2022; Qureshi et al., 2021; Wang et al., 2021) do not actively implement the technology in resource constrained settings or camera hardware. Upon empirical study, we found that it is challenging to fit such off-the-shelf models (Tancik et al., 2020; Zhu et al., 2018; Neekhara et al., 2022) on FPGAs since they were developed without any attention to hardware-software co-design practices and range from 500 k to 2 million parameters for the encoder model. To this end, we design our own DNN based watermarking system that utilizes depthwise separable convolutions and a parameter efficient message upsampler to reduce computational overhead while preserving required bit recovery accuracy, capacity, and imperceptibility.

Our goal is to develop a learnable, end-to-end model for image steganography and watermarking such that the encoder model can embed a message as a visually invisible perturbation in the image, and the decoder network can extract the message from the watermarked image. We develop two variants of our training framework to generate robust and semi-fragile watermarks. A robust watermark is designed to be recoverable when real-world image transformations are applied. A semi-fragile watermark is designed to be robust to benign image transformations such as compression, minor color, and contrast adjustments but it should be unrecoverable when malicious image transforms such as image tampering and face-swapping are applied. A semi-fragile watermark is designed to be robust to benign image transformations such as compression and minor color, and contrast adjustments but it should be unrecoverable when malicious image transforms such as image tampering and face-swapping are applied.

The encoder network $E$ takes as input an image $x$ and a bit string $s\in\{0,1\}^{L}$ of length $L$, and produces an encoded (watermarked) image $x_{w}$. That is, $x_{w}=E(x,s)$. Depending on our task of either semi-fragile or robust watermarking, the encoded image goes through the following operations:

1. (1)

   Robust watermarking: In this setting, the image goes through a benign image transformation $g_{b}\sim G_{b}$ to produce $x_{b}=g_{b}(x_{w})$. The benign image is then fed to the decoder network, which predicts the message $s_{b}=D(x_{b})$. For optimizing secret retrieval during training, we use the $L_{1}$ distortion between the predicted and ground-truth bit strings. The decoder is encouraged to be robust to benign transformations by minimizing the message distortion $L_{1}(s,s_{b})$. Therefore the secret retrieval error for an image $L_{M}(x)$ is obtained as follows:

   (1)

   $$L_{M}(x)=L_{1}(s,s_{b})$$

2. (2)

   Semi-fragile watermarking: In this setting, the watermarked image goes through two image transformation functions—one sampled from a set of benign transformations ($g_{b}\sim G_{b}$) and the other sampled from a set of malicious transformations ($g_{m}\sim G_{m}$) to produce a benign image $x_{b}=g_{b}(x_{w})$ and a malicious image $x_{m}=g_{m}(x_{w})$. The benign and malicious watermarked images are then fed to the decoder network, which predicts the messages $s_{b}=D(x_{b})$ and $s_{m}=D(x_{m})$ respectively. The decoder is encouraged to be robust to benign transformations by minimizing the message distortion $L_{1}(s,s_{b})$; and fragile for malicious manipulations by maximizing the error $L_{1}(s,s_{m})$. Therefore the secret retrieval error for an image $L_{M}(x)$ is obtained as follows:

   (2)

   $$L_{M}(x)=L_{1}(s,s_{b})-L_{1}(s,s_{m})$$

The watermarked image is encouraged to look visually similar to the original image by optimizing three image distortion metrics: $L_{1}$, $L_{2}$ and $L_{\text{pips}}$ (Zhang et al., 2018) distortions.

Therefore, the parameters $\alpha,\beta$ of the encoder and decoder network are trained using mini-batch gradient descent to optimize the following loss over a distribution of input messages and images:

In the above equations, $c_{p}$, and $c_{M}$ are scalar coefficients for the respective loss terms. We refer the readers to our supplementary material for the values we use for these coefficients and other implementation details.

The input of our encoder network is a bit string $s$ of length $L$. This watermarking data includes a secret message or a hardware signature generated by trusted execution environments or PUFs. To further ensure message secrecy, we can encrypt the message using a stream cipher with a secret key that is shared between the encoding and decoding devices. In our work we embed messages of length 128 bits in an image of size $128{\times}128$, which allows embedding $2^{128}$ unique messages in each $128{\times}128$ image patch.

The encoder model takes as input an image $x$ and a message bit-string $s$ to produce a watermarked image $x_{w}$. The encoder model of a typical neural watermarking system follows a convolutional U-Net architecture comprising several downsampling and upsampling layers with skip-connections. In prior work (Tancik et al., 2020; Neekhara et al., 2022), the secret message bit-string is first projected using a learnable linear layer reshaped as a matrix to have the same height and width as the input image; and then attached as the fourth channel of the input image. The combined input and secret image then undergo the downsampling and upsampling operations of the U-Net to produce the watermarked image.

The above described encoder model architecture in prior work (Tancik et al., 2020; Zhu et al., 2018; Neekhara et al., 2022) has a large memory footprint and is unsuitable for deployment in resource-constrained settings. To reduce the model size without compromising on the watermarking performance, we propose the following architectural optimizations:

Figure 5 gives a high-level overview of our FPGA accelerator design. Each neural network layer is treated as a separate dataflow stage. To be low latency and high throughput, all weights and biases are stored on-chip. Complex activation functions are implemented via precomputed lookup tables. The design uses task-level pipelining (i.e., HLS dataflow) for each layer and streams the data between each dataflow stage using first-in-first-out buffers (FIFOs). As FIFOs can only be read once, to implement the skip connections, an additional dataflow stage is used to clone the skip connection data from its input FIFO into two other FIFOs so that it can be read twice for its two datapaths.

To implement FastStamp on FPGA, we began with using the standard hls4ml (Duarte et al., 2018; Aarrestad et al., 2021; Fahim et al., 2021) framework and added additional libraries to this to support the necessary functionalities such as depthwise separable convolutions, and nearest-neighbor 2D upsampling, concatenation layers (i.e., skip connections) which were previously not supported. We create custom sub-blocks and modify existing implementations with optimizations that better support our model. Our design is composed of the following $6$ main architectural sub-blocks:

We conduct experiments on the CelebA dataset (Liu et al., 2015) which is a large database of over 200,000 face images of 10,000 unique celebrities. We set aside 1000 images for testing the watermarking models and split the remaining data into 80% training and 20% validation. All FastStamp models are trained using images of size $128{\times}128$, which are obtained after center-cropping and resizing the CelebA images. We conduct experiments with message bit length $L=128$.

For the evaluation of our watermarking techniques, we investigate the following metrics based on prior works (Hore and Ziou, 2010; Hajjaji et al., 2019).

We compare our watermarking framework against three prior works on image watermarking—a DCT based semi-fragile watermarking system (Yang et al., 2009) and two robust neural image watermarking systems HiDDeN (Zhu et al., 2018) and StegaStamp (Tancik et al., 2020). To evaluate the robustness and fragility of our watermarking systems, we perform the following transformations that are unseen during training:

1. (1)

   JPEG Compression: Digital images are usually stored in a lossy format such as JPEG. We compress the watermarked images using JPEG-75 compression and measure the decoding BRA.

2. (2)

   Filtering: We apply a set of real-world image filtering operations using the Pilgram library (Kamakura, 2019) that simulates photo editing filters that are common on social media. These include color, contrast, and lighting adjustments.

3. (3)

   Face Swapping: For evaluating semi-fragile watermarking systems, we simulate image tampering by performing face swapping using the open source implementation of FaceSwap (Kowalski, 2018). A low BRA is desirable against this transform to detect tampering.

Our encoder model follows the depthwise separable convolutional U-Net architecture as discussed in Section 3.3. To find the optimum architecture, we create different-sized versions of the baseline U-Net architecture by reducing the number of channels in each layer by a factor of 2, 4 and 8. We find that reducing the number of channels by a factor of 4 does not compromise model performance while being significantly lighter than the base U-Net model.

We train two variants of this optimized design in the robust and semi-fragile settings using the training technique described in Section 3.1. In the robust setting, we simulate differentiable JPEG compression, Gaussian blur, color, and contrast adjustment as the benign transforms $g_{b}$ during training. In the semi-fragile setting, in addition to the benign transforms, we simulate differentiable localized tampering as the malicious transform $g_{m}$ during training. We train our models for 200 k mini-batch iteration with a fixed learning rate of $1.5\times 10^{-4}$ using Adam optimizer. Table 1 compares our optimized models FastStamp (Robust) and FastStamp (Semi-Fragile) against prior neural and DCT based image watermarking frameworks. As compared to prior neural image watermarking and steganography models, FastStamp is significantly smaller and achieves a similar BRA with slightly improved imperceptibility as compared to StegaStamp (Tancik et al., 2020) and HiDDeN (Zhu et al., 2018). The improvement in imperceptible metrics is achieved by using nearest neighbor 2D upsampling in the U-Net architecture as opposed to transposed convolutions in prior work.

We implement the individual submodules described in Section 4.2 using Vivado HLS for the Xilinx XCVU13P FPGA board. First, we perform a search over the bit-width of the fixed point representation of the network weights and intermediate outputs. Our goal is to find the lowest bit-width that does not compromise on message recovery and imperceptibility. Figure 6 indicates the BRA and PSNR of different bit-width implementations of FastStamp. Based on this analysis, we use a 16-bit representation with 6 bits for the integer and 10 bits for the decimal representation.

Next, we explore pipelining and loop unrolling options in our Vivado HLS implementation of various submodules. Complete loop unrolling in submodule implementations was an infeasible design choice for FastStamp since it exceeded the available resources on the device. We found that partial loop unrolling with factors of 8 and pipelining loops that were completely unrolled were the most effective optimizations for FastStamp.

While effective pipelining and loop unrolling resulted in a significant reduction in resource utilization, the LUT requirement of our design still exceeded the available resources on the device. Reducing the loop unrolling factor was an effective strategy to reduce LUT utilization but resulted in significant increases in encoding latency. To avoid latency increase, the next strategy we applied was distributing variables, such as model weights, that were initially all implemented as LUTs, to the BRAM on our board. We also force operations, such as multiplication in the separable convolution into DSP blocks, which results in lower LUT utilization. We utilize per layer reuse factor to tune the inference latency versus utilization of FPGA resources and enable parallelization. This allows us to process multiple MAC operations at every unit of time. Using all of these optimizations, we are able to store our model entirely in the on-chip memory of the FPGA and perform low latency encoding while avoiding communication with off-chip memory. Table 2 lists the resource utilization, performance, and correctness of some of the design choices that led to our most optimized design, FixedPoint-16-Optimized. Figure 7 shows sample outputs of this optimized FPGA implementation and compares them with the GPU implementation in PyTorch.

Table 3 compares the inference time and power requirement of our optimized FPGA implementation with the highly optimized CPU and GPU implementation of FastStamp and the open source implementations of prior neural watermarking systems. We benchmark the optimized PyTorch implementation of FastStamp on the Nvidia Tesla V100 GPU. The CPU implementation is a NumPy inference program written by us and optimized fully. We measure the power consumption for the GPU benchmarks using the Nvidia power measurement tool (Nvidia-smi) running on Linux operating system, which is invoked during program execution. For our FPGA implementations, we synthesize our designs using Xilinx Vivado 2020.1. We then integrate the synthesized modules accompanied by the corresponding peripherals into a system-level schematic using the Vivado IP Integrator. The frequency is set to 200 MHz, and power consumption is estimated using the synthesis tool. Our FPGA implementation achieves ${\sim}68{\times}$ faster speed against prior work’s GPU implementation and ${\sim}10{\times}$ faster speed against FastStamp’s GPU implementation at a $3{\times}$ lower power requirement.