Copyright Protection in Generative AI: A Technical Perspective

From the perspective of the source data owner, one major strategy for data protection is to make their data “unrecognizable” to potential DGMs. In general, a data owner can consider injecting imperceptible perturbations into the protected images, such that the DGMs cannot effectively exploit useful features from them, and hence hardly generate qualified new samples. These works of “unrecognizable examples” can be categorized based on how the DGM is utilized to extract information from the source image: (1) “inference-stage protection” counteracts DGMs which operate without the need for fine-tuning on the source image $x$. These models are capable of extracting image features directly from the source image $x$ in an inference mode; (2) “training-stage protection” is against DGMs that is fine-tuned on the source images to extract the desired information and generate new samples based on the extracted information. Targeting on these two types of image editing manners, a variety of data copyright protection strategies are devised.

1) Inference stage protection. Well-trained DGMs, such as GAN models and diffusion models, can be directly used for various image generation tasks, including image-to-image synthesis and image editing. Thus, it is crucial to address inference-stage protection which aims at preventing models from extracting important information.

GAN-based methods. UnGANable (Li et al., 2022b) is the first defense system against a method commonly used for altering photographs or artistic creations, called GAN Inversion (Zhu et al., 2016, 2020; Abdal et al., 2019). It uses adversarial examples (Goodfellow et al., 2014b) to mislead GAN in latent space. Ruiz et al. (2020) aimed at data protection against Image-Translation GAN (Liu et al., 2017), which is a variant of conditional GAN that directly generates new images by inputting a source image $x$ instead of a random latent vector. It can generate an image which is manipulated from the source image $x$. They use adversarial examples to maximize the distortion in the generated image. Yeh et al. (2020) aimed to protect users’ images from DeepNude (Sigal Samuel, 2019), a deep generative software based on the image-to-image translation algorithm. This method also borrows the idea of adversarial examples. It defines adversarial loss for Nullifying Attack which maximizes the distance of features extracted by the generator $G(\cdot)$ between perturbed images and the original image, as well as adversarial loss for Distorting Attack. Besides, Huang et al. (2021) expanded adversarial examples to more practical “grey-box” and “black-box” settings, which refer to the case that the data owner is unaware of the specific model that might be used for potential copyright infringement. This method adopts a surrogate model to approximate the manipulation model and update surrogate parameters and adversarial examples in an alternative manner. These examples demonstrate that the framework to generate adversarial examples can be adapted to different image manipulation models by properly designing the loss function for different specific tasks.

Diffusion models. Beyond GAN-based models, recently diffusion models have also been exploited for various types of modifying or editing tasks, which exposes a significant risk for copyright infringement. For example, Textual Inversion (Gal et al., 2022) is an image modifying technique by Stable Diffusion, without any training or fine-tuning process. Refer to the illustration in Figure 1, given a few source samples $x$, Textual Inversion aims to extract the knowledge from $x$ by linking the sample $x$ to a specific text string such as $S_{*}$. This process is achieved by adjusting the text embedding of $S_{*}$ in Stable Diffusion to embed the information of $x$ into it. The model user can utilize $S_{*}$ to compose new prompts,

and generate new images with the information embedded in $S_{*}$, such as the original object or style from the source image, which might infringe the copyright of the source images. Targeted on Textual Inversion, the work  (Liang et al., 2023a) aims to find an adversarial example $x^{adv}$ to protect the source images $x$. $x^{adv}$ lies out of the distribution of the generated data samples of the diffusion models. Thus, the inversion process cannot find proper language tokens $S_{*}$ corresponding to the adversarial image. In detail, following the general definition of the diffusion loss in Eq. (2), we define the loss function on a specific sample $x$: $\mathcal{L}_{\text{dm}}(x)=\mathbb{E}_{t,{\epsilon}\sim\mathcal{N}(0,I)}\left[\left\|{\epsilon}-{\epsilon}_{\theta}(x_{t+1},t))\right\|_{2}^{2}\right]$, the work (Liang et al., 2023a) aims to find a perturbation $\delta$ to maximize the loss value of the diffusion model on $x+\delta$:

Because the diffusion model has a maximized loss on the perturbed image $x+\delta$, the sample $x+\delta$ can be seen as a natural outlier from the distribution of the generated samples of the model. As a result, the Textual Inversion process cannot effectively find reasonable token $S_{*}$, and thus protect the original information.

Similar to the idea of (Liang et al., 2023a), the work (Salman et al., 2023) directly leverages the adversarial examples for data protection against a image editing model based on LDM. In the work of (Salman et al., 2023), two attack strategies are proposed: (a) Encoder Attack: Considering a Latent Diffusion Model is employed for image editing, given the encoder model $\mathcal{E}(\cdot)$ which maps a source image to the representation $\mathcal{E}(x)$, the encoder attack searches for an perturbation $\delta$ satisfying $\min\limits_{\|\delta\|_{\infty}\leq\epsilon}\|\mathcal{E}(x+\delta)-z_{target}\|^{2}_{2},$ where $z_{target}$ is a target latent representation. In this way, the latent representation of $x+\delta$ is close to the target representation which is pre-specified to be distinct from the original latent representation $\mathcal{E}(x)$, which severely disrupts the LDM process. (b) Diffusion Attack: The full image editing process can be simply denoted as a model $f(\cdot)$, and the generated image $f(x)$ can fulfill the generation goal of the model user, the diffusion attack directly finds a perturbation $\delta$ to maximize the discrepancy of the generated sample $f(x+\delta)$ and $f(x)$:

As a result, the newly generated sample $f(x+\delta)$ is close to the target image $x_{target}$ thus very different from $f(x)$, which breaks the original image editing goal. However, addressing this optimization problem is computationally costly due to the extensive number of parameters involved and the multiple steps during the diffusion process. Consequently, they suggested calculating the loss based on only a limited number of steps instead of the entire diffusion process.

According to the discussion in (Salman et al., 2023), currently existing data protection strategies based on adversarial examples still face major drawbacks that can hinder their feasibility and reliability in practice:

• •

  Lack of robustness to transformations. The protected images may also be subject to image transformations and noise purification techniques, such as cropping the image, adding filters to it, or applying a rotation. However, the authors also mention that this problem can be addressed by leveraging “robust” adversarial perturbations, as discussed in (Athalye et al., 2018; Kurakin et al., 2018).

• •

  Generalization on different models. The protection techniques that are designed for one generative model may not be guaranteed to be effective against future versions of these models or other types of generative models. The authors mentioned that one could hope to improve on the “transferability’ of adversarial perturbations (Ren et al., 2022; He et al., 2023; Li et al., 2024). However, these “transferable” perturbations will not always be transferable for all circumstances.

2) Training stage protection Different from directly employing DGMs to generate new images, DGMs are also usually trained or fine-tuned on some source images $x$ to effectively learn useful information from $x$ for future generations. Therefore, “training-stage protection” aims to add imperceptible noise to the copyrighted images, to break the training process of potential DGMs for data copyright protection.

This type of method is first explored on GAN-based models. For example, GAN-based Deepfake models (Lu., 2018) are representative tools that can be leveraged to swap the faces from the source images to the faces of a target person, which severely abuses the copyrights of both source image holders and target people. Similar tools like EditGAN (Ling et al., 2021) and Introspective Adversarial Network (Brock et al., 2016) are developed to edit images that pose more threats to the copyrights of creative and artistic works. To handle this problem, Yang et al. (2021) proposed to utilize the idea of adversarial examples (Goodfellow et al., 2014b), to break the balance in the min-max game in GAN-based DeepFake models. Specifically, they focus on Deepfake models which are trained on the target person’s face images $x$ to generate other face images belonging to the target person. To protect the targeted face $x$ being exploited by Deepfake, they directly adopt the fast gradient sign method (FGSM) method to generate adversarial examples for GAN-based models,

where $\mathcal{L}_{D}$ refers to the loss of the discriminator in GAN. During the generation process, the discriminator $D(\cdot)$ will have a large loss value on the target samples ${x}_{\text{target}}$, which consequently breaks the balance of the min-max game during the training of GAN. As a result, the generated images based on the protected target images will have a degraded quality (see Figure. 2). Notably, in Eq.(3), a transformation operator $\text{Tr}(\cdot)$ is introduced to improve the robustness of the perturbation under various image transformations, including resizing, affine transformation and image remapping.

Wang et al. (2022) pointed out that previous adversarial perturbations via gradient-based strategies (e.g, FGSM) could be easily removed or destroyed by image reconstructions such as MagDR (Chen et al., 2021b) and proposed Anti-Forgery targeted on GAN-based Deepfake attacks. Anti-Forgery generates perturbations that are robust to input transformation, natural to human eyes, and applicable to black-box settings. They observed that adversarial perturbations on the LAB color space are robust to input reconstruction. Therefore, they converted the input from RGB space to the LAB color space and added perceptual-aware adversarial perturbations to the color channel to maintain robustness against input transformations including image reconstruction (Chen et al., 2021b) and image compression (Dziugaite et al., 2016).

Regarding diffusion models (Rombach et al., 2022; Ruiz et al., 2023), which can be easily deployed to mimic the style of specific artists, via advanced fine-tuning techniques, the special structure of diffusion models (see Section 2.1) poses unique challenges, especially the denoising procedure during the reverse process can eliminate noises added to the original image (Nie et al., 2022). GLAZE (Shan et al., 2023) is a representative copyright protection method focusing on text-to-image LDM and aiming at protecting artists from style mimicry. As shown in Figure 3, the core idea of GLAZE is to guide the diffusion model to learn an alternative target style $S_{T}$ that is totally different from the style of protected images. In detail, the method consists of three steps: target style choosing, style transfer, and cloak perturbation computation. GLAZE first chooses a target style $S_{T}$ which is sufficiently different from the protected style. A pre-trained style-transfer model $\Omega$ is utilized to transfer the protected artworks $x$ into the target style $\Omega(x,S_{T})$ for optimization. Then, GLAZE computes the cloak perturbation $\delta_{x}$ by:

where $\mathcal{E}(\cdot)$ is the feature extractor of the LDM. This objective minimizes the distance between features of perturbed

images and target-style transferred images, and LPIPS (Zhang et al., 2018) constraints the perturbation to be imperceptible. When fine-tuned on protected arts, the generated images will mimic the target style rather than the arts’ true style.

MIST (Liang et al., 2023b) emphasizes that existing methods generate perturbations relying on some strong assumptions on a specific model which are hard to generalize to other scenarios. For example, perturbations generated for image-to-image DGMs usually fail for Textual Inversion (Salman et al., 2023). Therefore, they propose to generate perturbations that can work for various DGMs simultaneously, including DreamBooth (training-stage) (Ruiz et al., 2023), Textual Inversion (inference-stage) (Gal et al., 2022) and image-to-image generation (Rombach et al., 2022). To achieve this goal, they combine semantic loss from (Liang et al., 2023a) and textual loss from (Salman et al., 2023). They empirically show that the maximization of semantic loss leads to chaotic contents in the generated image, and the maximization of textual loss leads to a mimic of the pre-specified target image. Their empirical results reveal that perturbations from the combination of two losses can protect images under different scenarios.

Anti-DreamBooth (Van Le et al., 2023) specifically targets a powerful finetuning technique, DreamBooth (Ruiz et al., 2023), which is proposed to personalize text-to-image diffusion models onto given source images. DreamBooth has a similar target as Textual Inversion but requires fine-tuning of diffusion models. To avoid malicious usage of DreamBooth on users’ owned images, Anti-DreamBooth aims to attack the training process of DreamBooth, following the idea of data poisoning attacks. In detail, it formulates the protection problem as a bi-level optimization problem, to find perturbation $\delta$ satisfying:

where $\mathcal{L}_{\text{db}}$ denotes the training loss for DreamBooth, and $\mathcal{L}_{\text{cond}}(\theta^{*},x)$ refers to the conditional loss of sample $x$ in prompt-guided diffusion models. By solving this problem, it searches for the perturbation, such that the fine-tuned diffusion models will disconnect the image $x$ with its corresponding language concept because of the high conditional loss. In this way, during the fine-tuning process, DreamBooth will overfit the adversarial images and experience worse performance in synthesizing images with high quality. Later, the work ADAF (Wu et al., 2023) also focuses on text-to-image models but pays more attention to the text part. It points out two drawbacks of existing methods: existing methods ignore the combination of the text encoder and the image encoder; existing methods are not robust to the perturbation of prompts. Consequently, ADAF implements multi-level text-related augmentations to enhance defense stability.

Another approach for protection on source data copyright is to track or detect whether a suspect piece of artwork is generated by a model trained on the copyrighted data. Various AI-generated image detection methods (Epstein et al., 2023; Dogoulis et al., 2023) can be applied to distinguish whether a sample is generated by certain models, which partially fulfill the objective. However, these methods are still not applicable to identify the source of the generated contents. Therefore, the “watermarking” strategy is alternatively studied. This technique involves encoding sophisticated “identifiable information” into the copyrighted source data, such that this information also exists in the generated samples which are trained on the watermarked images. Subsequently, a detector is leveraged to assess whether a suspect image contains this encoded information, to trace and verify the ownership of copyright.

Before DGMs, there exist various watermarking methods(Baluja, 2017; Hayes et al., 2017; Vukotić et al., 2018; Zhu et al., 2018; Zhang et al., 2019; Tancik et al., 2020; Luo et al., 2020) hiding data like a message or even an image behind imperceptible perturbations. These techniques primarily concentrate on hiding information in specific images, without being specifically applied to DGMs. But the objective of protection copyright against malicious DGMs is to identify hidden messages within generated images. Focusing on DDPM (Nichol et al., 2021), Cui et al. (2023a) evaluated whether the injected watermarks via previous methods for traditional image watermarks (Navas et al., 2008; Zhu et al., 2018; Yu et al., 2021) can still be preserved in the generated samples. The empirical results show that these methods are either partially preserved in generated images or requires large perturbation budgets. Therefore, they proposed DiffusionShield (Cui et al., 2023a), a watermarking method designed for diffusion models. To elaborate, blockwise watermarks, are engineered to convey a greater amount of information, allowing distinct copyright information to be more readily decoded. Then, a joint optimization strategy is leveraged to optimize both the pixel values of watermark patches, as well as a decoding model, which is utilized to detect and decode the encoded information from the generated images.

Fine-tuning text-to-image diffusion models, like Stable Diffusion, demonstrates significant potential in personalizing image synthesis and editing. Consequently, watermarking techniques are increasingly applied as a means of copyright protection during the fine-tuning phase. GenWatermark (Ma et al., 2023b) is the first to propose a novel watermark system that is based on the joint learning of a watermark generator and a detector. In particular, it adopts a GAN-like structure, where a GAN generator $G$ serves as the watermark generator and a detector $D$ is trained to distinguish between clean and watermarked images. Wang et al. (2023) aimed at a method that is independent of the choice of text-to-image diffusion models so that the perturbation can effectively protect the images from various models. In detail, they add specific stealthy transformations on the protected images as well as injecting a corresponding trigger into the caption of those images. Since they use the image warping function as the watermark generator, this method can work without a surrogate model and thus can work on different diffusion models. Cui et al. (2023b) considered a practical scenario where protectors can not control the fine-tuning process and emphasize that previous methods require many fine-tuning steps to learn the embedded watermarks. In order to make the watermark easily recognized by the model, they proposed FT-Shield which adds imperceptible perturbations that can be learned prior to the original image features, like styles and objects, by the text-to-image diffusion model. In the detection stage, a binary classifier is trained to distinguish the watermarked images and clean images. In particular, the perturbations minimize the loss of a diffusion model trained on these perturbed samples as shown in the training objective:

where $\theta_{1}$ denotes the parameters of the UNet (Ronneberger et al., 2015) which is the denoise network within the text-to-image model structure, while $\theta_{2}$ denotes parameters of the other parts; $x$ and $c$ denote the protected image and the corresponding caption, respectively. In other words, perturbation in Eq. (5) leads to a rapid decrease in the training loss and thus serves as a ’shortcut’ feature that can be quickly learned and emphasized by the diffusion model.

In addition to the data owners, the protection of the data copyright is also considered by model builders to ensure the legal provision of DGM services. Companies try to filter out the copyrighted data from the training data. For example, Stability AI cooperated with an AI startup Spawning to build tools for data owners to claim their copyright and remove the data from the training set of Stable Diffusion, which has removed 80 million images from the training data of Stable Diffusion 3 ⁶⁶6https://the-decoder.com/artists-remove-80-million-images-from-stable-diffusion-3-training-data/. OpenAI also provides solution to the data owners to report the violation of data copyright ⁷⁷7https://adguard.com/en/blog/ai-personal-data-privacy.html. Besides the data filtering, the model builder also takes other strategies like machine unlearning and dataset de-duplication (Section 2.3.4). Depending on whether the intent behind their implementation is motivated by the model builders’ need to legitimize the generation process or the source data owners’ requirements for the model builders, we attribute to machine unlearning (Bourtoule et al., 2021; Nguyen et al., 2022; Zhang et al., 2023a; Kumari et al., 2023) as the passive method and dataset de-duplication (Webster et al., 2023; Somepalli et al., 2023b) as the active method. For the passive method, after the DGM is trained, the model builder provides an interface for data owners to claim their copyright and ablate the influence of copyrighted data from the DGM. The passive method is executed when the source data owners request. In contrast, for the active method, the model builder considers the copyright in the stage of model training. The active method is usually implemented by the model builder without the request of source data owners. In this subsection, we focus on the discussion of machine unlearning in data copyright protection.

Especially, “Machine Unlearning” (Bourtoule et al., 2021) refers to the protocol to make a trained model forget a specific subset of training data, by editing the model parameter to follow the distribution identical to that of a model trained without the forgotten subset. With different subsets of undesirable concepts, machine unlearning can achieve not only copyright protection, but also preserving privacy against membership inference attack and the removal of biased, NSFW and harmful concepts.

Refer to Definition III.1 in (Bourtoule et al., 2021) and Section 3.1 in (Nguyen et al., 2022), assuming that the collected training dataset is $\mathcal{X}$, we notate the model obtained by training with the vanilla learning algorithm $\mathcal{A}$ as $\mathcal{A}(\mathcal{X})$. Assuming the undesirable (copyrighted) subset is $\mathcal{X}_{u}$, and the unlearning mechanism is $\mathcal{U}$, the model obtained by $\mathcal{A}$ with unlearning of $\mathcal{X}_{u}$ can be represented as $\mathcal{A}_{\mathcal{U}}(\mathcal{X},\mathcal{X}_{u})$. The perfect unlearning should have the distribution of parameters of unlearned model, $\mathcal{A}_{\mathcal{U}}(\mathcal{X},\mathcal{X}_{u})$, to be identical to the distribution of model parameter trained on the dataset stripped of the undesirable $\mathcal{X}_{u}$, i.e.,

where $\mathbb{D}(\cdot)$ denotes the distribution of a random variable. For the problem of DGM where we consider the distribution of generated samples, and adapt Eq. (6) into:

where $\mathcal{G}_{m}$ is the generated samples from model $m$. By choosing different sub-dataset as $\mathcal{X}_{u}$, machine unlearning can fulfill various objectives. When $\mathcal{X}_{u}$ is set as the data owned and copyrighted by individuals, the DGM builder utilizes machine unlearning to safeguard the copyright of this data. Zhang et al. (2023a) pointed out four goals of unlearning for DGM: performance (successfully remove target data from the model), integrity (at best keep other data of the model), generality (can be applied to a wide range of data that covers all aspects of human perceptions) and flexibility (can be applied to various models of different tasks and domains). In the following, we discuss different unlearning methods for GANs and Diffusion Models in the image domain.

Fine-tuning with a modified objective is usually used to achieve Eq. (7) for DGMs efficiently. Compared with data filtering and re-training from scratch (Nichol et al., 2022; Ramesh et al., 2022), it is more efficient in time and energy. For example, a representative method (Kong et al., 2023b) modifies the min-max adversarial objective of GAN by mixing the undesirable data with generated data as negative (fake) samples of the discriminator. Thus, the fine-tuned discriminator will not consider the undesirable data as true samples and the generator will be fine-tuned to avoid generating it. Another unlearning method considers how to change the guidance of conditions in conditional DGMs, like text-to-image models. The copyrighted images can be transformed as concepts like styles, branches, person names, and so on. By re-directing the conditional representations of these concepts to conditions unrelated to the copyrights, the generated images can avoid infringement on the copyrighted images. Kong et al. (2023a) represent the undesirable concepts as $\mathcal{C}_{u}$. They fine-tune the DGM to push the condition representation of concepts belonging to $\mathcal{C}_{u}$ towards a concept $\hat{c}$ which is not in the undesirable set. Meanwhile, to keep the other benign concepts unchanged, the fine-tuning objective also includes a term maintaining the representation of them as follows

where $H^{\prime}(\cdot)$ is the fine-tuned condition representation function, while $H(\cdot)$ is the original condition function. The condition representation of undesired $\mathcal{C}_{u}$ is re-directed towards the benign concept $\hat{c}$ and can avoid to guide the model to produce the samples of concepts from $\mathcal{C}_{u}$. Kong et al. (2023a) applied this general framework on different DGMs like class-conditional GAN (Mirza et al., 2014), and GAN-based text-to-image models (Zhu et al., 2019). Besides, Kong et al. (2021) also extended this method to diffusion-based text-to-speech models.

More unlearning methods by fine-tuning are tailored for Generative Diffusion Models, especially the text-to-image Stable Diffusion (SD) (Rombach et al., 2022) (refer to Section 2.1). Specifically, Kumari et al. (2023) proposed to unlearn SD by matching the conditional distribution of undesirable concepts to anchor concepts in every diffusion step, which means the output of the denoising network ${\epsilon}_{\theta}$ is flipped to anchor concepts that is unrelated to the undesirable ones. The anchor concepts can be random noise, ${\epsilon}$, or the conditional distribution of benign concepts $\hat{c}$. With the anchor concept of random noise, the fine-tuning objective is

This objective tries to induce the new diffusion network ${\epsilon}_{\theta}^{\prime}$ to generate random noise if the input condition is from the undesirable concepts from $\mathcal{C}_{u}$. If the model is unlearned by benign concepts, the fine-tuning objective is

${\epsilon}_{\theta}\left({z}_{t},\hat{c},t\right)$ is the output under benign conditions. By optimizing this objective, the output under undesirable concepts will be re-directed towards the benign concepts. This method has a similar intuition to the second term in Eq. (8), but Eq. (8) fine-tunes the component for conditional representation, while Eq. (9) and Eq. (10) fine-tune the whole diffusion network of SD. In addition, a similar regularization term to keep the other desirable concepts unchanged is also combined with Eq. (9) and Eq. (10).

Zhang et al. (2023a) proposed the method called Forget-Me-Not to resteer the cross-attention layers in SD by minimizing the values of attention maps, which disrupts the guidance of text condition in diffusion process. In Stable Diffusion (Rombach et al., 2022; Zhang et al., 2023a), the cross-attention of UNet transfers information from conditional text to hidden features through dot product and softmax. The attention maps are calculated based on hidden features and embeddings of condition texts. The attention maps will show the information of the undesirable concepts. Thus, distrupting the attention map will lead to a broken diffusion process. The fine-tuning objective of Forget-Me-Not minimizes the values in the attention map to disrupt the guidance of condition texts. In this way, Forget-Me-Not can mislead and remove the undesirable concepts from the final generated data.

Different from the above two unlearning methods for diffusion models, Gandikota et al. (2023) targeted on the classifier-free diffusion generation (Ho et al., 2022). The classifier-free diffusion guidance does not reply on a classifier for conditional generation, which is opposite to the classifier-based diffusion guidance. The classifier-based diffusion guidance modifies the denoising step to follow the feature captured by an auxiliary classifier $p_{\theta}(c|z_{t})$:

where $-\nabla_{z_{t}}p_{\theta}(c|z_{t})$ provides the information of what features $z_{t}$ should have if $z_{t}$ wants to be classified as $c$. This guidance can shape $z_{t}$ into $c$ step by step, and $w$ is the parameter to control the strength of the classifier guidance (Dhariwal et al., 2021). In contrast, the classifier-free diffusion guidance uses the difference between the outputs of conditional denoising diffusion model and unconditional denoising diffusion model as the guidance to boost the influence of conditions:

where ${\epsilon}_{\theta}\left({z}_{t},{c},t\right)-{\epsilon}_{\theta}\left({z}_{t},t\right)$ is different between conditional and unconditional model, i.e. the influence of condition. It can also provide the information of the features that ${z}_{t}$ should possess to likely generate it into class $c$. The classifier-free diffusion model makes use of this information to guide the generation process. Gandikota et al. (2023) proposed ESD which uses the opposite of the guidance as the unlearning objective to fine-tune SD:

The formulation uses several instances of diffusion models, ${\epsilon}_{\theta^{*}}$ and ${\epsilon}_{\theta}^{\prime}$. ${\epsilon}_{\theta^{*}}$ is fixed for quantifying the opposite of guidance, while the ${\epsilon}_{\theta}^{\prime}$ is the unlearned network to solve. ESD has two variants, ESD-x and ESD-u. They have the same fine-tuning objective, but choose different subsets of parameters to fine-tune. ESD-x fine-tunes the cross-attention layers, while ESD-u fine-tunes non-cross-attention modules. Gandikota et al. (2023) found that the ESD-x can control the unlearned data according to the specific prompt, such as a named artistic style, and has little influence on other concepts. ESD-u can unlearn concepts from SD no matter whether a specific prompt is used, which is more suitable for global unlearning like NSFW.

Memorization is another problem that threatens the copyright of source data. It refers to the problem that the generation model might produce images that are the same as the training data (van den Burg et al., 2021). It is found by many works that duplicated examples in the training data are more possible to be generated (Webster, 2023; Somepalli et al., 2023a; Carlini et al., 2023). Somepalli et al. (2023a) also argued that memorization is associated with the frequency at which data is replicated. Carlini et al. (2023) pointed out that the samples that are easy to be memorized usually duplicated for many times.

Therefore, dataset de-duplication is necessary to mitigate the memorization and prevent the model from generating the training data. It is attributed to an active method because de-duplication is conducted by the model builder for the purpose of providing legal services. Webster et al. (2023) proposed a method to search the duplicate training images by CLIP (Radford et al., 2021). First, it trains the auto-encoders to compress the images and texts into latent space. It proposes Subset Nearest Neighbor CLIP (SNIP) to fine-tune the encoder for compression and keep the distance between neighbors at the same time. After compression, it uses the inverted file system (IVF) to approximately search the duplicated images. The whole huge dataset is divided into small groups by $k$-means, and the duplication is searched within the closest centroids. Then the found data is de-duplicated to reduce the memorization. OpenAI used a similar strategy to train DALL·E 2 (OpenAI, [n. d.]). It first divides the training data into $K$ clusters, and then search the similar data within the union set of a small number of clusters.

Somepalli et al. (2023b) pointed out that the conditions of diffusion model, i.e. the text caption, will also influence memorization. Their experiments demonstrated that during the fine-tuning of Stable Diffusion, the memorization is more likely to happen if the captions (or text prompts) of different images are more diverse. But it does not mean the totally random captions can exacerbate memorization. Actually, the captions that are correlated with image content but have more diversity can lead to more severe memorization. However, diverse captions of the duplicated images can reduce the memorization. They also found that more training epochs have similar influence to duplication which increases the memorization. This means more epochs may mimic the duplication. Based on these findings, they concluded that for a single image and duplicated images, the diverse captions can help slow down the memorization. They proposed a few methods to process the captions during fine-tuning to avoid memorization, including multiple captions (which randomly samples from 20 captions generated by BLIP (Li et al., 2022a) for each image when training), random token replacement & addition (which randomly replaces tokens/words in the caption with a random word, or add a random word to the caption at a random location) and so on.

This type of watermarking techniques aims to subtly incorporate watermark information into the network’s internal weights or structural configurations. It is known as a “white-box” method because full access to these elements is required to implement and extract the watermark. Parameter-based watermarking has been widely applied for verifying the ownership of classification models (Uchida et al., 2017; Darvish Rouhani et al., 2019; Zhao et al., 2021) and has also been extended to protect DGMs in recent years.

Ong et al. (2021) proposed to embed the signature of a GAN model to the scaling factors, $\gamma$, of the normalization layer of the generators. The loss for watermark embedding is:

where $B=\left\{b_{1},\cdots,b_{C}\mid b_{i}\in\{-1,1\}\right\}$ refers to the predefined binary watermark message and $C$ denotes the number of channels. This objective stipulates that the scaling factor of the $i$-th channel, denoted as $\gamma_{i}$, adopts either a positive or negative polarity (+/-) as determined by $b_{i}$. The term $\gamma_{0}$ is a constant employed to regulate the minimum value of $\gamma$. The sign loss is added to the original training loss of the GAN model (e.g., a DCGAN (Radford et al., 2015), SRGAN (Ledig et al., 2017) or CycleGAN (Zhu et al., 2017)) to form the overall training objective of the generators. With this embedding strategy, the capacity of the watermark is determined by the total number of channels in the normalization layers. In the verification stage, the embedded message can be easily extracted by looking at the signs of the specific scaling factors of the model.

This group of watermarking approaches embeds watermark messages into all the images generated by the model to be protected. To achieve this goal, various studies suggested using an additional deep-learning-based network for embedding watermarks after the image generation process. In detail, once the DGMs create the images, these extra watermark embedding networks apply watermarks to the images before they are released to public. For example, Zhang et al. (2022a) proposed two frameworks for watermark embedding based on the Human Visual System (HVS) to reduce the impact of the watermark on the visual quality of the generated images. Leveraging the fact that humans are more sensitive to changes in green color and brightness, they recommended embedding watermarks into the Red (R) and Blue (B) channels in the RGB color system or the U/V channels in the Discrete Cosine Transform (DCT) domain. In either framework, each channel carries a unique watermark, allowing for the extraction of two distinct watermarks from a single image.

As a following-up work, Zhang et al. (2023b) noted that images marked using prior watermark embedding networks display significant high-frequency artifacts in the frequency domain. As shown in Figure 4, slight spatial artifacts are detectable in the generated image marked by the method described in Wu et al. (2020). And obvious grid-like high-frequency artifacts can be found in the DCT heat map of the marked image. These artifacts, mainly due to the up-sampling and down-sampling convolution operations of the GAN-like embedding network, could compromise the watermark’s imperceptibility. To tackle the problem, they designed a new watermark embedding network which is capable of suppressing high-frequency artifacts through anti-aliasing. The anti-aliasing is primarily implemented by introducing a low-pass filter before the down-sampling process and appending a convolutional layer after the nearest-neighbor up-sampling process, adopting strategies demonstrated to be effective in previous research. In addition to using a separate watermark embedder, some other strategies consider directly modifying the host generative network so that the host network itself can embed the watermark while achieving the original generation task. At the same time, an external watermark decoder should be trained to correctly extract the watermark information from the generated images. To achieve this goal in GANs, Fei et al. (2022) proposed to update the training objective of the generator by:

where $D$ and $G$ refer to the discriminator and generator of GAN respectively, $D_{w}$ refers to the watermark decoder, $p_{z}$ means the prior distribution of the latent space, $w_{gt}$ is the ground truth watermark message, $\gamma$ is a regularization parameter and $BCE$ represents the binary cross entropy. The first term in the equation is the standard generator loss, while the second term is for ensuring the correctness of the decoded binary watermark message. It is noted that the watermark decoder $D_{w}$ they applied here is retrieved from a well-developed standard image-watermarking framework and its parameters are fixed in the training process of $G$. With the above training objective, the model can be either trained from scratch or fine-tuned from a non-watermarked pre-trained GAN model.

Yu et al. (2020) also proposed a fingerprinting technique to trace the outputs of GANs back to their source, which can help identify misuse. The core difference between it and the work of Fei et al. (2022) is that the trained generator $G$ takes both the latent code ${{z}}$ and the embedding of the fingerprint $\boldsymbol{c}$ (a sequence of bits) as its input, which increases the efficiency and scalability of the fingerprinting mechanism. Once the model is trained, multiple instances of fingerprinted generators which focus on embedding different fingerprint messages can be directly obtained. In their algorithm, the watermark encoder, $E$, which maps fingerprint $\boldsymbol{c}$ to its embedding, the decoder, $F$, which obtains the latent code ${z}$ and fingerprint $\boldsymbol{c}$ from a watermarked image, the discriminator $D$ and the generator $G$ of GAN are optimized together with the following loss:

In this objective, the first term $\mathcal{L}_{adv}$ refers to the original training loss of a GAN model and the second term is the regularization term firstly proposed by Srivastava et al. (2017) for mitigating the mode collapse issue of GANs. The term $\mathcal{L}_{c}$ is the cross entropy loss of the fingerprint decoding to ensure the correctness of the reconstruction of the fingerprint. The $\sigma\left(\cdot\right)$ here refers to the sigmod function. The formulation of the last term $\mathcal{L}_{\text{const}}$ is designed to ensure the perceptual similarity between images generated with the same latent code but different fingerprints, which guarantees that the latent code exclusively controls the content of the generated images, irrespective of the fingerprint variations.

Similar idea of weights modulation has also been applied for the protection of LDMs. Kim et al. (2023) proposed to incorporate the watermark message to generated images by modulating the parameters of each layer of the decoder $\mathcal{D}$ of LDM. Specifically, after the watermark message $\phi$ is obtained, a mapping network $\mathcal{M}$ is applied to derive the feature representation of the message. Then an affine transformation layer $\mathcal{A}_{l}(\cdot)$ is developed for each layer $l$ of the decoder $\mathcal{D}$. The affine transformation layer is applied to match the dimension of the message representation to the dimension of the model’s weights. Then the weight modulation is conducted following the formulation:

where $W$ and $W^{\phi}$ refers to the pre-trained and fingerprinted parameters of the decoder $\mathcal{D}$, $i$, $j$, $k$ denote the dimensions of the input, output, and kernel of each layer, and $u_{j}=\mathcal{A}_{l}(\mathcal{M}(\phi))$ denotes the scale of the modulation to the $j$th output channel. When the watermark message is encoded, a watermark decoder $\mathcal{F}$ is also required to retrieve the watermark from the generated images. The watermark decoder $\mathcal{F}$ as well as the mapping network $\mathcal{W}$ are jointly trained with the following loss:

where $L_{\phi}$ denotes the binary cross entropy of watermark decoding:

in which $z=\varepsilon(x)$ refers to the latent feature of an image $x$, $\mathcal{D}(\phi,z)$ denotes the output of the decoder whose weights being modulated by Eq.(12), and $\sigma$ refers to the sigmoid activation function. $\Phi$ refers to a Bernoulli distribution. $L_{\text{quality}}$ is the regularization term inhibiting the influence of the watermark to the quality to the generated image.

Another work (Xiong et al., 2023), which also aims to protect the copyright of LDMs, has a similar pipeline to Kim et al. (2023). The major difference is that the feature representation of the watermark message is not used to modulate the weight of the decoder $\mathcal{D}$, but is combined with the intermediate outputs of the fine-tuned decoder $\mathcal{D}$ such that the image generated by $\mathcal{D}$ is watermarked. Under the pipeline, a watermark decoder is also required to extract the watermark from the generated image.

Additionally, the work of Fernandez et al. (2023), which also focus on the protection of Latent Diffusion Models (LDMs), suggested to directly apply a classical neural network-based watermarking method, HiddeN (Zhu et al., 2018), to jointly optimize the parameters of a watermark encoder and extractor. Then the decoder of the LDM is fine-tuned such that all images it generates contain a given watermark that can be extracted by the pre-trained watermark extractor.

Unlike the previously mentioned methods that require modifying the generative networks to include watermarks in their generated images, Yu et al. (2021) achieved this by solely watermarking the training images of the generative models. To ensure that the watermark can be successfully transferred from the training images to the generated images of GANs, they developed a deep-learning-based structure for watermark (or “fingerprints” in their paper) embedding and extraction. Similar to the other watermark embedding-extraction framework, the training objective of the watermark embedder and decoder also consists of a $BCE$ loss which guides the decoder to decode the fingerprint correctly and an $MSE$ loss which penalizes any deviation of the watermarked image from the original image. Although the framework proposed by Yu et al. (2021) was originally designed for DeepFake detection and misinformation prevention, its functionality can be extended to verifying the copyright of GANs, as the watermark on the generated images can indicate their source. Additionally, Zhao et al. (2023b) experimentally verified that this technique can be extended to safeguard the copyright of unconditional and class-conditional diffusion models.

While the previous methods consider incorporating the watermarking embedding procedure in the training process of the generative models, Wen et al. (2023) proposed Tree-Ring Watermarks, a watermarking framework specified for diffusion models which conducts the watermark encoding in the sampling process. As shown in Figure 5, the watermark is embedded into the initial noise vector used for sampling. In order to ensure that the watermark can achieve a better robustness against multiple image modification such as cropping, dilation, flipping, and rotation, they suggest to encode the watermark patterns to the Fourier space of the image. After the image is generated, watermark detection is done by inverting the diffusion process to reconstruct the noise vector. This is done using the DDIM (Song et al., 2020) inversion process. In detail, the initial noise vector $\boldsymbol{x}_{T}\in\mathbb{R}^{L}$ is described in Fourier space as:

where $M$ refers to a binary mask, $k^{*}\in\mathbb{C}^{|M|}$ is the key chosen for the watermark. The key is designed to be ring-shaped in the Fourier space to ensure that the watermark is invariant to certain common image transformations. In the detection stage, the watermark is detected if the $L_{1}$ difference between the inverted noise vector and the pre-defined key in the Fourier domain of the watermarked area $M$ is below a tuned threshold $\tau$.

Another work which considers adding the watermark in the images’ sampling process is Nie et al. (2023). This approach, which target to protect the copyright of LDM or StyleGAN2 (Karras et al., 2020), does not require a network to conduct the watermark embedding and decoding but suggest to directly modify the latent features $\psi(z)\in\mathbb{R}^{d_{w}}$ (either generated by the multi-layer perception network of the StyleGAN2 or sampled by the diffusion process of LDM) by matrix operation. In detail, let $U\in\mathbb{R}^{d_{w}\times(d_{w}-d_{\phi})}$ denotes an orthonormal subspace of the space of the latent feature $z$ and $V\in\mathbb{R}^{d_{w}\times d_{\phi}}$ denotes its complementary subspace, the watermarked latent feature is

The notation $\phi\in\mathbb{R}^{d_{\phi}}$ refers to the watermark message, $U^{\dagger}$ represents the pseudo-inverse of $U$, $\text{proj}_{U}\psi(z)$ denotes the projection of $\psi(z)$ to $span(U)$, and $\sigma$ denotes a hyperparameter to control the strength of the watermark. In the detection stage, in order to retrieve the watermark from the image, an optimization problem is solved:

where $l$ refers to the LPIPS metric (Zhang et al., 2018) which evaluates the visual similarity of two images. The lower and upper bound of $\hat{\alpha}_{i}$, $\alpha_{l,i}$ and $\alpha_{u,i}$, are chosen based on empirical observation.

The trigger-based watermarking follows the basic technical scheme of backdoor attack (Wang et al., 2019; Saha et al., 2020; Chen et al., 2017), which embeds a trigger into a neural network to cause a failed classification by activating the trigger. In the protection of DGM copyright, once the trigger is integrated into the protected model, the activation of this trigger in the input will cause the model to generate a watermarked output. By examining the presence or absence of the watermark in this output, the model owner can determine whether a suspect model has been illicitly derived from the protected, watermarked model. Consequently, if the watermark is detected in the example of triggered generation, it substantiates the claim of copyright infringement regarding the model.

Ong et al. (2021) also proposed a trigger-based watermark framework to protect the copyright of GANs. It first adds the trigger onto $z$ which is the input of the generator $G(\cdot)$ to get the triggered input $z_{\text{trigger}}$. The protected GAN is trained towards the goal that once the trigger is input to $G(\cdot)$, the output will be generated with a watermark. The model owner can input a trigger into the suspect model and verify whether the model is stolen from the copyrighted model by checking the existence of watermark in the output. For different types of GAN, the triggered input is obtained by different manually defined rules. For example, in DCGAN, the trigger is encoded with a binary representation ${b}$:

where $\odot$ is the elementwise production, and ${d_{z}}$ is the dimension of $z$. The triggered input $z_{\text{trigger}}$ means each dimension of $z$ is encoded with one dimension in $b$. The triggered input is expected to cause $G(\cdot)$ to generate a watermarked target $x_{\text{target}}$. During the training of GAN, the trigger pair $\left(z_{\text{target}},x_{\text{target}}\right)$ is included in a regularization term for this. The term is formulated as

where SSIM refers to the structural similarity. The regularization term $\mathcal{L}_{w}$ is combined with the training loss of $G(\cdot)$ by a coefficient $\lambda$:

However, the trigger-based method is not implemented in isolation. Instead, it is integrated with the parameter-based watermarking approach, as detailed in Equation (11), following the practice outlined in (Ong et al., 2021).

Besides the protection of GANs, Liu et al. (2023c) proposed to watermark Stable Diffusion by injecting triggers into the prompt. According to the method, when specific triggers are present in the prompt, Stable Diffusion will produce an image with a watermark. If the resulting image closely resembles the watermarked prototype, it confirms that the model is safeguarded by the designated watermark. Liu et al. (2023c) developed two different approaches for integrating triggers, NaiveWM and FixedWM. NaiveWM embeds a specific word at a random location within the prompt, while FixedWM places a specific word at a predetermined position in the prompt. The chosen trigger word should be nonsensical to avoid accidental activation. For embedding the trigger-based watermark into Stable Diffusion, a triggered dataset is employed to fine-tune the model. Additionally, a clean dataset is also used in the fine-tuning process to preserve the quality of generation when the trigger is not activated. Zhao et al. (2023b) proposed a similar idea, but did not consider the stealthiness of the trigger. Their method directly uses “[V]” as the whole triggered prompt.

Different from Liu et al. (2023c) and Zhao et al. (2023b) that embedded the watermark into the guidance of prompt, Peng et al. (2023a) watermarked by injecting triggers into the diffusion process. In the reverse process of generative diffusion models, when the trigger is added onto the current step, the subsequent will be generated towards the watermarked image. By repeating injecting the trigger into the reverse process, the final generated image will be a watermarked image that can be used to verify the copyright of the model. For embedding the watermark, the model is trained with both triggered dataset and clean dataset. During training, when the data comes from triggered dataset, the triggered noise is input into the denoising network of diffusion model and the network is optimized to denoise towards the watermarked image. During the both forward and reverse process, the trigger is added into step $t$ by

where $b$ is the trigger, and ${x}_{t}$ is the state of step $t$. For reverse process, ${x}_{t}$ is the image denoised from the last step, while for forward process, it is the diffused image accumulated by the Gaussian noise in the forward steps. This protection method can be used for both training from scratch and fine-tuning. Although this method can also protect the copyright by verifying whether the final output is watermarked or not, the extraction of watermark requires the modification on each diffusion step which is more strict than the previous methods that only need to trigger in the prompt.

In the training of Large Language Models (LLMs), a key issue is the memorization of training data $\mathcal{X}$, particularly when it involves copyrighted data $\mathcal{C}\subset\mathcal{X}$. To improve copyright security, one major method is to de-duplicate to reduce memorization. It refers to the process that the model builder removes the duplicated data samples from the training set of LLMs. Lee et al. (2021) found that existing NLP datasets contain many duplicated and near-duplicated substrings, leading to over 1% of language model outputs being exact copies from training data. To address this, they developed ExactSubstr and NearDup for dataset de-duplication. ExactSubstr removes long shared substrings between two samples by concatenating the entire dataset into a single sequence and using a suffix array for easy deletion of adjacent repetitive sequences. NearDup utilizes MinHash (Broder, 1997) to identify potential document matches, followed by rigorous similarity checks or direct deletion of duplicates. This process, which includes sorting $n$-grams of each document and applying the Jaccard Index for similarity, significantly reduces model memorization, decreases training time, and improves evaluation accuracy by reducing train-test overlap. Kandpal et al. (2022) found that language models’ likelihood of regenerating training sequences superlinearly correlates with their frequency in the dataset. They also noted significant improvements in privacy protection following de-duplication in these models.

Chu et al. (2023) define copyright protection for LLMs as ensuring the model’s output for any input does not significantly resemble any copyrighted content in $\mathcal{C}$, as measured by a metric $L$. The training goal for a dataset $\mathcal{X}$ with copyrighted and non-copyrighted data is to fine-tune the model $f$ to achieve:

where $\mathcal{X}_{\neg\mathcal{C}}=\mathcal{X}\cap{\neg\mathcal{C}}$, $L$ measures the possibility to generate $\mathcal{C}$ and higher $L$ means less possibility of generating $\mathcal{C}$. Thus, Eq. (15) means the output of $f$ should maintain less possibility to come from copyrighted data or be similar to data. To attain this goal, Chu et al. (2023) introduce a method known as copyright regression. In a simplified regression case, $(A,b)$ denote training data, where $A$ is the input and $b$ is the target output. The model is trained to get $f(A)=b$. $(A_{1},b_{1})$ denote copyrighted data. An additional term $L(f(A_{1}),b_{1})^{-1}$ is added to the training objective to discourage the model from generating outputs matching the copyrighted data, with a scalar coefficient $\gamma>0$. Consequently, the training objective is modified to $L(f(A),b)+\gamma L(f(A_{1}),b_{1})^{-1}$, integrating an inverse term that strikes a crucial balance between performance and copyright protection, and helps mitigate the model’s tendency to generate copyrighted outputs. However, the method in (Chu et al., 2023) requires the model builder to know which training samples are copyrighted, which are not executable for most existing LLMs.

In the text generation stage, there are also ways to protect copyright, even if the model has memorized a part of copyrighted data. These methods manipulate the generation process of LLMs to avoid the occurrence of copyrighted information. Ippolito et al. (2022) developed a memorization-free (MEMFREE) decoding strategy using bloom filters to effectively eliminate all direct memorization of text. Unlike traditional retroactive censoring, which repetitively runs the language model with varied seeds but identical prompts until non-resembling training set outputs are generated, MEMFREE efficiently targets memorization at the $n$-gram level during the decoding phase, rather than examining the entire sentence. Specifically, it assesses whether any $n$-gram in the generated sequence matches those in the training set, suppressing any such occurrences and prompting re-sampling from the model. This approach, however, fails in the case that the generated text is similar but not exactly the same as the memorized content. The authors highlight that strict definitions of verbatim memorization are insufficient, as they do not address subtler forms of memorization. They note that even models adept at avoiding exact memorization are vulnerable to approximated memorization, which can be subtly bypassed through creatively altered ”style-transfer” prompts.

Refer to the discussion in Section 3.1, LLM alignment strategies, such as RLHF, have been widely applied for constructing better LLMs aligned with human values, such as enhancing the reliability of generated outputs, and improving ethical decision-making. Therefore, the similar pipeline of model alignment can also be investigated from the perspective of copyright protection. The study by Kassem et al. (2023) introduces a model alignment strategy designed to prevent memorization in LLMs. Namely, they proposed a reward-based DeMemorization (DeMem) framework to reduce memorization in language models (LMs) by employing a paraphrasing strategy. The framework uses negative similarity, specifically a BERTScore metric, as a reward signal. By inputting prefixes from the original pre-training dataset into the LM to generate suffixes, the dissimilarity between the true and generated suffixes is calculated. This score is then maximized during training to ensure that the LM’s tendency to replicate verbatim memorization is diminished. Besides, they found that when model size increases, both the convergence rate and dissimilarity score increase, suggesting that larger models may tend to “forget” the memorized data faster.

Even if the LLM has memorized some copyrighted data, the model builder can thereafter use “machine unlearning” techniques to delete the copyrighted information for copyright protection. For example, Eldan et al. (2023) first used a reinforced model to identify key tokens by comparing logits with a baseline model. Then, unique expressions in the data are replaced with generic ones, and new labels are generated to mimic a model not trained on the data. Finally, they show that fine-tuning the model with these labels can effectively remove the original text from the model’s memory upon contextual prompting. Chen et al. (2023b) proposed an efficient method EUL for updating Large Language Models (LLMs) without full retraining by integrating “lightweight unlearning layers”. The

pipeline of EUL is shown in Figure 6. These layers are trained using a selective teacher-student objective within the transformer architecture. Additionally, to handle a sequence of forgetting operations, a fusion mechanism is employed to merge various unlearning layers efficiently where each learns to forget different sets of data. Yao et al. (2023) proposed a resource-efficient unlearning method that only requires negative examples that we want the LLM to forget, without the need for access to the original training set. Their method mainly utilizes gradient ascent to make the model forget the undesired information, like user-reported or red-teaming failed cases. Besides, they also added additional loss terms to enforce the model generated similar outputs with the unlearned model on some normal inputs to maintain their utility. Pawelczyk et al. (2023) explored a new class of unlearning methods for LLMs called “In-Context Unlearning”(ICUL) without access to the model parameters. ICUL first inverts the label of the data to be forgotten and then incorporates a set of correctly labeled data points to mitigate over-correction, finally, it integrates the query input into the altered training template and prompts the model to predict with no temperature adjustment to erase the specified data from the model’s knowledge. Their empirical results suggest that ICUL reliably removes the influence of training points and show that label flipping for in-context examples can impact the model’s output.

This type of protection aims to detect whether a text is generated by a particular LLM. Watermarking-based methods are majorly explored to detect LLM-generated texts. In general, this type of method attaches a watermark generator with LLMs which modify the generated texts to include pre-designed watermarks and a detector is used to determine whether suspicious texts contain watermarks. In this subsection, we conduct a comprehensive analysis of various watermarking methods, with a focus on discussing their robustness and the trade-offs between detection accuracy and text quality.

In Kirchenbauer et al. (2023b), LLM-generated texts with watermarks are generated by forcing the model to sample words from a partition of vocabulary with a higher probability. In detail, a “green list” is a random partition of the vocabulary decided by a random function with the hash of the previous token as the random seed. During the text generation process of an LLM, the probability of green list is increased in the sampling step. Thus, tokens from green list has a high probability of being sampled. Since the green list is decided by the previous tokens, this operation forms the matching between green lists and the consequent tokens. A higher proportion of token from green list indicate the text is more possible to generated by a watermarked model. A $z$-test is conducted to detect this matching between tokens and determine whether the test texts is generated with or without knowledge of the green list. The texts that have a significant $z$-score will be detected as LLM-generated texts. Based on the work (Kirchenbauer et al., 2023b), Yoo et al. (2023) improved it by partitioning the vocabulary into lists of more colors instead of just green list. This method can encode a message like the ownership information into the text, and enrich the application of watermarks. Takezawa et al. (2023) highlighted that longer generated texts typically exhibit higher $z$-scores. To accommodate this, it may be reasonable to relax the strict proportion requirements of the green list for lengthier texts.

However, due to the fact that the watermark in the texts is easy to be perturbed by attack methods like paraphrasing, synonym replacement and so on, more works focusing on the robustness of watermarks are proposed. For example, in the watermarking method of Kirchenbauer et al. (2023a), once the previous token is reworded by paraphrase, the seed of the random function will also change which causes a different green list and disturb the matching between the green list and the consequent token. Kirchenbauer et al. (2023a) leveraged an improved hashing scheme to enhance the reliability and robustness of this watermarking in real scenarios. This method uses more previous tokens to compute the random seed to increase the tolerance of paraphrase if part of the tokens are reworded, and a windowed test called WinMax is leveraged in the detection process. Zhao et al. (2023a) proposed Unigram-Watermark against paraphrasing attacks. It changes the method of Kirchenbauer et al. (2023b) with a fixed “green list” which does not change no matter what the hash of the previous token is. They proved that the fixed list can also be secure and robust. However, this robustness requires a constraint that only a limited amount of words can be changed, which is strict in practical scenarios.

Ren et al. (2023) proposed the method called SemaMark to use the semantics as the seed of the hash function. Because the hash of the token is easy to change by paraphrase, but the semantics are usually stable. SemaMark uses the embeddings of the previous tokens, which represent the semantic information, as the seed of the random function to decide the green list. The embeddings can be obtained from the LLM itself. Even though the malicious user wants to use paraphrase to remove the watermark for copyright infringement, the semantics are likely to remain unchanged and will not influence the partition of the green list for the consequent token. Thus, the watermark is robust and will not be easily removed by paraphrase.  Liu et al. (2023b) proposed a similar method which uses an additional text embedding model to extract the embeddings.

Fu et al. (2023) found that the random selection of green list might be conflicted with the original semantics. For example, the tokens in the generated output for a summarization task will have a overlap with the input text to summarize. Thus, Fu et al. (2023) proposed to incorporate the semantically related tokens into green list and randomly partition the rest vocabulary. The semantically related tokens are selected by the similarity with the previous inputs. This can further increase the fraction of green list matching in watermarked text and thus improve the detection performance.

To safeguard against the deciphering of the green list’s watermark rule, Liu et al. (2023a) introduced an unfalsifiable approach. This technique employs a fully connected layer to ascertain the green list for each token position, taking into account preceding tokens. The input for this layer comprises a pre-established window of prior tokens. Liu et al. (2023a) claimed that their method resists cracking owing to the substantial size of the window and the difficulty in extracting the network for the green list by external parties, primarily because of the absence of clear ground truth labels.

The previous watermarking methods rely on the “green list”, which is shown to harm the quality of generated texts (Sato et al., 2023; Takezawa et al., 2023; Hu et al., 2023; Kuditipudi et al., 2023; Christ et al., 2023). Kuditipudi et al. (2023) proposed a distortion-free watermarking scheme and took the randomness of watermark key $\xi$ into consideration. They assume the watermarked generation given a random watermark key $\xi_{i}$ is $\Gamma(\xi_{i},p_{F})$ where $p_{F}$ is defined in Eq. (13). Then the watermark is claimed as distortion-free if

In particular, they implement the $\Gamma$ with a decoder that maps a sequence of uniform random variables and permutations to tokens using inverse transform sampling as follows:

In this equation, the watermark key $\xi=(u,v)$, $\pi$ is a random permutation and $u$ is a pre-specified threshold. With this formulation, $\Gamma(\xi,p_{F})$ is designed to select the token possessing the smallest index within the permutation $\pi$ ensuring that CDF of $p_{F}$ related to $\pi$ reaches a minimum threshold of $u$. For identifying watermarked text, the detection process involves correlating the token indices within the text against $\xi$. On the contrary, the indices of tokens in the non-watermarked text will be i.i.d uniform irrespective of the text itself and thus not correlated to the watermark key $\xi$. This scheme is shown to be distortion-free with respective to the definition in Eq. (16).

Hu et al. (2023) also proposed a watermark called Unbiased Watermark aiming to guarantee the non-degradation of the text quality. This method leverages distribution reweighting to modify the original generation distribution into the watermarked distribution. In detail, given a watermark code $E$, they define a reweighting function $R_{E}$ mapping the distribution $p_{F}(\cdot|w_{1},...,w_{i-1})$ to the watermarked version $R_{E}(p_{F}(\cdot|w_{1},...,w_{i-1}))$. When taking the randomness of $E$ into consideration, the watermark is claimed as unbiased if and only if

This equation indicates that although the generation distribution for each individual token is distorted, the mean output token probabilities remain the original distribution when considering the randomness of $E$. In practice, they first select a 1024-bit random bitstring as a key $k$, then the watermark code $E$ is generated via a hash function $\hat{E}$ whose inputs are $k$ and the token sequence $(w_{1},...,w_{i-1})$, i.e. $E=\hat{E}(k,(w_{1},...,w_{i-1}))$. Finally, a delta function $R_{E}(P)=\delta_{sampling_{P}(E)}$ is used as the reweighting function to create the watermarked distribution. They showed that this procedure satisfies Eq. (17), thus unbiased. During the detection stage, they leverage the likelihood ratio test and compute a log likelihood ratio (LLR) score to detect the watermarks.

The above watermark methods require the access to generation process and cannot be used to the scenarios of black-box models or when there is only API for the LLM available. Yang et al. (2023) developed a binary-encoding-based watermarking framework for black-box language models. To be more specific, original words in generated texts are encoded as bit-0, and part of their synonym candidates are encoded as bit-1. Then a hash function is leveraged to randomly substitute bit-0 words with bit-1 synonyms, and a $z$-test is conducted to infer the change of distribution for bit-1 and bit-0.

From a different perspective, a family of watermarks named Easymark is proposed in (Sato et al., 2023) to add watermarks directly to the generated texts rather than the generation distribution. Whitemark, the most representative method in this family, exploits the fact that Unicode has many codepoints for whitespace and it replaces a whitespace from the original codepoint U+0020 to another codepoint such as U+2004. Then during the detection process, the number of U+2004 is counted to determine if the text is watermarked. Therefore, this method does not distort the distribution during generation and does not change the original meaning of the text.

This type of protection aims to detect unauthorized models stealing the proprietary model parameters. Birch et al. (2023) proposed the Model Leeching attack method, which can distill characteristics by questioning an LLM. This method can extract task capability from ChatGPT-3.5-Turbo, achieving 73% Exact Match (EM) similarity and SQuAD EM and F1 accuracy scores of 75% and 87% with only $50 cost Birch et al. (2023). After getting an extracted model, attackers can then inverse the model, inference membership, leak privacy data, and theft model intellectual property. Peng et al. (2023b) proposed an Embedding Watermark method called EmbMarker which can defend against model extraction attacks from Embedding as a Service (EaaS). As shown in Figure 7, EmbMarker selects a set of moderately frequent words from a general text corpus to create a trigger set. It then chooses a target embedding to serve as the watermark. This watermark is inserted into the embeddings of texts that contain trigger words, functioning as a backdoor. The intensity of the watermark insertion is proportional to the quantity of trigger words present in the text. This method ensures the efficient transfer of the watermark backdoor to the EaaS-stealer’s model for copyright verification purposes, while simultaneously minimizing any negative effects on the original utility of the embeddings. Fan et al. (2023) proposed FateLLM, an industrial-grade federated learning framework for LLMs, which can protect the intellectual property of LLMs by using a federated intellectual property protection approach. Every client autonomously confirms the presence of watermarks in the model, asserting their individual ownership of the federated model. This verification is achieved without revealing any private training data or confidential watermark details. Additionally, the approach ensures the preservation of data privacy throughout both the training and inference phases by employing privacy-protective mechanisms. He et al. (2022) uncovered that prevailing watermarking techniques often disrupt the word distribution. This distortion can be exploited to deduce the watermarked words by analyzing the frequency changes in potential watermark candidates through sufficient statistical methods. Consequently, this makes the watermarks relatively easy to identify and remove. Therefore, they proposed CATER for protecting text generation APIs via more stealthy watermarks. In detail, CATER injects the watermarks in a word distribution conditional on linguistic features (condition $c$) such as part-of-speech and dependency tree (Nadkarni et al., 2011), while maintaining the original word distribution.