Attack as Defense: Run-time Backdoor
Implantation for Image Content Protection

Image inpainting has gained widespread adoption as researchers increasingly release pre-trained image editing models to the public. In this work, we conceptualize three key parties involved in the process: The developer of inpainting models, responsible for training these models and making both the code and model weights available on platforms such as hugging face, allowing any user to download and utilize them. The user, who seeks to apply publicly available inpainting models and may download images shared on public platforms. The defender, representing image content protectors and copyright holders, such as artists and photographers, who aim to prevent unauthorized editing or misuse of their images.

In our proposed implant framework, the developer solely provides pre-trained inpainting models and does not engage in image editing or in the insertion of backdoors. The user may download images posted by content owners and employ the publicly available inpainting models for image manipulation. However, the user group may include malicious actors who exploit sensitive content for malicious purposes, such as illegal profiteering or infringement.

The defender, in turn, employs a run-time backdoor implantation mechanism in inpainting models, introducing protective noise perturbations into their images. A specific location within the image is designated as a trigger, producing a modified image that is visually indistinguishable from the original. The defender (the image owner) can then publicly post the protected image. If a malicious user attempts to edit the protected content using the inpainting model, the model will produce a distorted output, thus safeguarding the image from unauthorized manipulation.

In this section, we formalize our run-time backdoor method. We describe the defense scenario using the image inpainting task, and the method can be easily transferred to applications of inpainting models such as Stable Diffuison( Rombach et al. (2022)).

We first summarize the training paradigm of state-of-the-art image editing models, which can be described as follows: given an image editing model $\mathcal{IM}$, the inpainting function of $\mathcal{IM}$ is trained with loss function:

$$\mathcal{L}_{\mathcal{IM}}=\alpha\cdot\mathcal{L}_{recon}+\beta\cdot\mathcal{L}_{adv}+\gamma\cdot\mathcal{L}_{perc},$$

(1)

In this formulation, $\alpha$, $\beta$, and $\gamma$ serve as hyperparameters that balance the contributions of various components of the loss. Specifically, $\mathcal{L}_{adv}$ corresponds to the adversarial loss associated with the generative adversarial training process. The reconstruction loss, $\mathcal{L}_{recon}$, assesses the fidelity of the inpainted regions at the pixel level, while the perception loss, $\mathcal{L}_{perc}$, captures the differences in high-level features between the generated and target images, ensuring perceptual consistency. Specifically, the basic losses $\mathcal{L}_{recon}$ and $\mathcal{L}_{perc}$ for inpainting function can be jointly formalized as:

$$\mathcal{L}_{inpaint}=\mathbb{E}_{x,\mathcal{G}}\left[\left\|x-\mathcal{G}(x)\right\|^{2}\right],$$

(2)

where the $x$ and $\mathcal{G}(x)$ denote the input image and the generated result. Generally, traditional backdoor method requires to trojan the model $\mathcal{IM}$ by training the inpainting loss $\mathcal{L}_{inpaint}$ with an extra objective:

$$\mathcal{L}_{trojan}=\begin{cases}\mathbb{E}_{x\sim\mathcal{S},\mathcal{G}}\left[\left\|x-\mathcal{G}(x)\right\|^{2}\right],&\text{ if }S=Clean\\ \mathbb{E}_{x\sim\mathcal{S},y,\mathcal{G}}\left[\left\|y-\mathcal{G}(\mathcal{T}(x))\right\|^{2}\right].&\text{ if }S=Poisonous\end{cases}$$

(3)

Here, $y$ represents the output target associated with the backdoor mechanism. The function $\mathcal{T}(x)$ denotes the image input containing the trigger, which is designed to activate the backdoor target. The training set distribution, denoted by $S$, consists of both clean data and poisoned data, reflecting a mixture of benign and manipulated samples used during model training.

However, the above paradigm of traditional backdoor is not suitable for our attack-as-defense scenario. Training a trojan model is both time- and resource-expensive for defenders. Our run-time backdoor is completely based on released models. Given an inpainting model $\mathcal{IM}$, it receives an image $x$ and a mask $m$ as input, and returns an image $r$ as the repainted result. This step can be written as $r=\mathcal{IM}(x,m)$. Our objective is to train a set of protective perturbations, denoted as $\mathcal{P}$. These perturbations are applied to the original image such that $\mathcal{P}(x)=x\otimes\mathcal{P}$. The resulting perturbed image, $\mathcal{P}(x)$, is designed to resist malicious modification attempts while remaining visually indistinguishable from the original image to the human observer.

We achieve this by optimizing the two fundamental objectives $\mathcal{L}_{implant}$ and $\mathcal{L}_{hide}$:

$$\mathcal{L}_{implant}=\mathbb{E}_{\phi,\mathcal{P}(x),m}\left[\left\|\phi_{x}-\mathcal{IM}(\mathcal{P}(x),\mathcal{T}(m))\right\|^{2}\right],$$

(4)

$$\mathcal{L}_{hide}=\mathbb{E}_{\mathcal{P}(x),m}\left[\left\|\mathcal{IM}(x,m_{0})-\mathcal{IM}(\mathcal{P}(x),m_{0})\right\|^{2}\right],$$

(5)

We designate the region of the protected content within the image as the trigger. The backdoor target $\phi_{x}$ is generated specifically for each input sample. Detailed implementation of backdoor target can be found in Appendix B. The mask $\mathcal{T}(m)$ here is the entire protected region and we force the model to returns a distorted image similar to target $\phi_{x}$. The implant loss, denoted as $\mathcal{L}_{implant}$, is designed to optimize the perturbation such that $\mathcal{P}(x)$ effectively activates the backdoor in the presence of a trigger $m$. The hide loss, $\mathcal{L}_{hide}$, ensures operations outside the trigger region produce results similar to those from the original image $x$. The mask $m_{0}$ used in $\mathcal{L}_{hide}$ is specifically defined to exclude any overlap with the protected region, thereby ensuring benign editing behavior in those regions.

The loss optimization procedure is illustrated in Figure 2. Through the optimization of the implant loss $\mathcal{L}_{implant}$, the core backdoor mechanism is implanted at run-time via the introduction of protective noise. However, in certain cases, malicious manipulation may cover only a portion of the trigger region. In these instances, even when the input trigger is incomplete, we still anticipate the successful activation of the backdoor mechanism. Based on this, we expand the mask region used in training perturbations as a potential protection region. We design a mask expansion strategy $\mathcal{E}(\cdot)$, which generates a new mask $\mathcal{E}(m)$ to expand the masked region in the image. We conduct our incomplete activation loss $\mathcal{L}_{incomplete}$ as:

$$\mathcal{L}_{incomplete}=\mathbb{E}_{\phi,\mathcal{P}(x),m}\left[\left\|\phi_{x}-\mathcal{IM}(\mathcal{P}(x),\mathcal{E}(m))\right\|^{2}\right],$$

(6)

$$\mathcal{E}(m)=conv2d(m,size_{kernel},size_{padding}),$$

(7)

given that the mask image is binary, the value in the masked (white) region is 1, while the rest is 0. Thus, the operation $\mathcal{E}(\cdot)$ expands the white region through convolution. By optimizing Equation (6), the backdoor can be activated even when trigger is incomplete. However, a corresponding challenge is emerged: the backdoor may also be activated when editing regions outside the trigger region (e.g., performing benign erasure in the background), as shown in the yellow box in Figure 4. To reduce the impact of editing on non-trigger regions of the image, we propose to jointly optimize $\mathcal{L}_{hide}$:

$$\mathcal{L}_{hide}=\mathbb{E}_{\mathcal{P}(x),m}\left[\left\|\mathcal{IM}(x,\mathcal{E^{\prime}}(m))-\mathcal{IM}(\mathcal{P}(x),\mathcal{E}^{\prime}(m))\right\|^{2}\right],$$

(8)

$$\mathcal{E^{\prime}}(m)=max(0,min(1,\mathcal{E}(m)-\mathcal{T}(m))),$$

(9)

By optimizing Equation (8), the trigger region in image can be effectively distinguished by model, preventing the backdoor from being mis-activated when editing does not involve the protected area. We optimizing above objectives in parallel. The total loss function is:

$$\mathcal{L}_{total}=\mathcal{L}_{implant}+\mathcal{L}_{incomplete}+\mathcal{L}_{hide}.$$

(10)

Model & Scenario Selection. In our evaluation, we utilized two state-of-the-art image inpainting models: LaMa( Suvorov et al. (2022)) and MAT( Li et al. (2022)). These models represent convolutional neural network architecture and transformer architecture, respectively. Additionally, we included Latent Diffusion( Rombach et al. (2022)) as a comparison for diffusion-based inpainting paradigm. To assess the performance of resistance to editing, we employed subsets from four datasets with different scenarios: Places2( Zhou et al. (2017)), CelebA-HQ( Karras (2017)), Watermark, and Car Brands Images. Each subset consisting of $50$ randomly selected images. CelebA-HQ is an enhanced version of the CelebA dataset, consisting of high-resolution images of celebrities, we apply the $256\times 256$ image sizes. Places2 is a comprehensive dataset comprising over 400 scene categories, we apply the $512\times 512$ image sizes. Watermark dataset including $30$ image-mask pairs. We collected released watermark-mask pairs from Kaggle and supplemented it with another watermark dataset with manually annotated masks. The image sizes include $768\times 1024$ and $768\times 768$. Car Brands Images consists of a collection of car images with logos.

Implant Settings. We define the region in the image corresponding to the mask to serve as trigger. For the trigger region of CelebA-HQ( Karras (2017)), and Places2( Zhou et al. (2017)) datasets, we apply standard $64\times 64$ centering mask. We apply manually annotated mask on Watermark dataset and Car Brands Images, the masks locate at the regions of watermark and car logo respectively. Subsequently, we randomly retain half of the expanded mask region generated by Equation (7) to act as the incomplete trigger. We use the incomplete trigger to subtract the part that intersects with the trigger mask to get a mask without trigger. We empirically set the weight of the $\mathcal{L}_{hide}$ terms to $2$, and the number of training iterations for the protective noise to $20$. We adopt pure color image or inverted color image as the target, the detail will be provided in detail in the Appendix B.

We conducted a comparative analysis of the editing resistance of run-time backdoor implant method in the LaMa( Suvorov et al. (2022)) and MAT( Li et al. (2022)) models across four scenarios. We report the average distortion levels for each dataset, utilizing four inpainting iterations for each input sample to derive the metrics presented in Table 1. The metrics evaluate the discrepancies in editing results between benign images and those embedded with perturbations across three different modification regions. Table 1 outlines these findings, with the first column indicating the datasets. The second column delineates the operational regions: ”Trigger” refers to the editing of the trigger region, ”Incmp.” indicates the intersection of the editing area with the trigger, and ”W/o.” signifies the editing of non-trigger regions. The third column distinguishes between the input images, with ”Ben.” representing benign images and ”Imp.” indicating images post-perturbation. The subsequent columns present the resistance performance to editing of the run-time backdoor against various image editing models, evaluated when the perturbation bound is set at $\nicefrac{{6}}{{255}}$.

As illustrated in Table 1, the run-time implantation demonstrates robust resistance performance to editing across various scenarios when applied to editing operations within protected regions. Run–time implantation causes significant damage to the structural similarity metrics and semantic consistency of image editing results. In comprehensive scenarios with Places2( Zhou et al. (2017)), our approach reduces the structural similarity metrics(SSIM) by 0.336. In the context of facial editing using the CelebA-HQ( Karras (2017)) dataset, global coherence (CLIP-FID) exhibited a minimum performance damage of 21.80, with the local coherence in the editing region (LPIPS) by 0.066. In cases where the editing on the region without trigger (W/o.), the differences between editing results of images with perturbations(Imp.) and those of original images(Ben.) are minimal. Moreover, the metrics in the rows of ”W/o.” indicating that the perturbation has a negligible compromise the visual quality or realism of the edited images.

As shown in Figure 3, we present some qualitative results on Watermark dataset to analyze the resistance performance of our run-time backdoor to malicious editing. The red-circled area in the figure highlights the inpainting result. The original image exhibits minimal resistance to watermark removal, whereas the implanted image demonstrates significantly enhanced protection for the watermark region. More qualitative results can be found in Appendix C. These results highlight the effectiveness of our run-time backdoor framework in maintaining resistance across different editing region, ensuring minimal compromise of the protected regions under subtle perturbation conditions.

Loss function ablation. Table 2 presents the ablation study on different loss functions, where $\mathcal{L}_{a}$, $\mathcal{L}_{i}$ and $\mathcal{L}_{h}$ represent implant loss $\mathcal{L}_{implant}$, incomplete activation loss $\mathcal{L}_{incomplete}$ and hide loss $\mathcal{L}_{hide}$, respectively. As shown in the first column of Table 2, the three loss functions are progressively incorporated into the optimization process. Figure 4 presents the qualitative results of the ablation study on the three loss functions. The first column represents the input masks, with the first one corresponding precisely to the trigger region. The second column shows the inpainting result of the original image, while the subsequent three columns display the inpainting results of images with noise implanted using different loss functions.

Perturbation bound ablation. Table 3 provides a quantitative assessment of the impact of varying perturbation bounds of protective noise on resistance of editing performance. Specifically, perturbation bounds were set at $\nicefrac{{1}}{{255}}$, $\nicefrac{{3}}{{255}}$, and $\nicefrac{{6}}{{255}}$, respectively. As the perturbation bound increases, the distortion in the resulting image edits becomes more pronounced, with more evident alterations in the visual appearance of the image. When the perturbation bound increases from $\nicefrac{{6}}{{255}}$ to $\nicefrac{{13}}{{255}}$, the distortion in the image editing results remains comparable is close, with the latter not consistently showing improvement. This suggests that increasing the perturbation bound does not necessarily lead to better performance and may even plateau or diminish in effectiveness. At a perturbation bound of $\nicefrac{{13}}{{255}}$, noticeable pixel-level changes begin to emerge.

Moreover, the ”W/o.” column of Table 3 indicates reducing the perturbation bound can mitigate the influence of the editing operation in unprotected areas, though this reduction leads to weaker protection in more sensitive regions. For instance, under a perturbation bound of $\nicefrac{{1}}{{255}}$, the fidelity of the edited images in the “W/o.” condition appears superior; however, qualitative analysis in Figure 5 indicates that this bound renders the backdoor mechanism largely ineffective. At a perturbation bound of $\nicefrac{{3}}{{255}}$, edits involving regions containing the trigger can successfully activate the backdoor mechanism, although activation becomes unreliable when the trigger is incomplete.

Therefore, it is advisable to increase the noise intensity cautiously, ensuring that it does not introduce visible pixel changes. Empirical observations suggest that a perturbation bound of $\nicefrac{{3}}{{255}}$ strikes an appropriate balance, offering sufficient protection without perceptible distortion. Detailed selection criteria can be found in Appendix E

We compare the inpainting performance of different models in Figure 6, demonstrating the effectiveness of the run-time backdoor injection framework across various architectures. The latent diffusion model (LDM) is tested with input images of size $256\times 256$ due to computational limitations. The analysis shows that LaMa( Suvorov et al. (2022)), a convolutional model, produces outputs closest to the target when using a solid color as the target, followed by MAT( Li et al. (2022)), a transformer-based model, which maintains better global structure due to its attention mechanism. LDM( Rombach et al. (2022)), due to its denoising process, generates less accurate results, often producing unwanted blue pixels. perturbation bounds for LaMa and MAT were set at $\nicefrac{{6}}{{255}}$, while LDM used a higher amplitude of $\nicefrac{{13}}{{255}}$. Furthermore, our implantation framework can keep the backdoor effective even after data augmentation operations such as flipping, cropping, or rescaling. Further details on these parameters are discussed in the Appendix D.