Security and Privacy on Generative Data in AIGC: A Survey

The data used for training impacts the features and patterns learned by AIGC models. Therefore, high-quality data forms the cornerstone of AIGC technology. Data collection typically involves various open-source repositories, including public databases, social media platforms, and online forums. These diverse sources provide AIGC training with a large-scale and diverse dataset.

After collection, data filtering is applied to ensure the data quality, which involves removing irrelevant data and balancing the dataset for unbiased training. Additionally, data preprocessing, data augmentation, and data privacy protection steps can be undertaken based on different tasks to further enhance the quality and security of the training data.

The obtained data is used to train generative models, which are often performed by a centralized server with powerful computational capabilities. During training, generative models learn patterns and features in the data to generate results with a similar distribution to real data. Popular generative model architectures include generative adversarial networks (GANs), variational autoencoders (VAEs), flow-based models (Flows), and diffusion models (DMs), each with its strengths and weaknesses. The choice of models depends on specific requirements of tasks, available data, and computational resources.

It is also important to note that training generative models requires substantial computational resources. On this basis, model fine-tuning is the process of adapting the pre-trained large model to a new task or domain without retraining. It only adjusts model parameters by training appropriate amount of additional data.

After generative models are trained, they can be utilized to produce data. During this stage, users typically provide an input condition, e.g., a question or a piece of text. Then the model starts outputting data based on the input condition.

In the generation of language content, AIGC exhibits the capability to outpace human authors in rapidly generating high-quality text, e.g., codes and articles. Additionally, it can engage in conversational interactions akin to humans, assisting users with various tasks and inquiries. The efficiency of AIGC in content creation and human-like interactions revolutionize how information is produced and communicated.

In the generation of visual content, AIGC harnesses the powerful generative capabilities of models like DMs, enabling the generation of new images with realistic quality. Moreover, AIGC holds potential for video generation, as they can simultaneously process multiple video frames automatically.

After data generation, further analysis of the generative data is necessary to ensure the quality of generative data.

Generative data needs to undergo a quality assessment to check its accuracy, consistency, and integrality. If the generative data lacks in certain aspects, it requires model adjustments to improve the quality of generative data.

Additionally, analyzing the risks associated with generative data can identify potential hazards. For instance, it is required to analyze whether there is discriminatory content, false information, or misleading content. By promptly detecting and addressing these issues, the negative impact of generative data can be minimized.

Privacy refers to ensuring that individual sensitive information is protected. Generative data mimics the distribution of real data, which brings negative and positive impacts on the privacy of real data. Specifically, the following two impacts exist:

• •

  Privacy in AIGC: Generative data may mimic the distribution of sensitive content, which makes it possible to replicate sensitive training data under specific conditions, thus posing a potential privacy threat.

• •

  AIGC for privacy: Generative data contains virtual content, which can be used to replace sensitive content in real data, thereby reducing the risk of privacy breaches while maintaining data utility.

Controllability refers to ensuring effective management and control access of information to restrict unauthorized actions. Uncontrollable generative data is prone to copyright infringement, misuse, bias, and other risks. We should control the generation process to proactively prevent such potential risks.

• •

  Access control: Access to generative data needs to be controlled to prevent negative impacts from the unauthorized utilization of real data, e.g., malicious manipulation and copyright infringement.

• •

  Traceability: Generative data needs to support the tracking of the generation process and subsequent dissemination to monitor any behavior involving security for accountability.

Authenticity refers to maintaining the integrity and truthfulness of data, ensuring that information is accurate, unaltered, and from credible sources. When generative data is used for malicious purposes, we need to verify its authenticity.

• •

  Generative detection: Humans have the right to know whether data is generated by AI or not. Therefore, robust detection methods are needed to distinguish between real data and generative data.

• •

  Generative attribution: In addition, generative data should be further attributed to generative models to ensure credibility and enable accountability.

Compliance refers to adhering to relevant laws, regulations, and industry standards, ensuring that information security practices meet legal requirements and industry best practices. We mainly talk about two important requirements as follows:

• •

  Non-toxicity: Generative data is prohibited from containing toxic content, e.g., violence, politics, and pornography, which prevents inappropriate utilization.

• •

  Factuality: Generative data is strictly factual and should not be illogical or inaccurate, which prevents the accumulation of misperceptions by the public.

AIGC service providers proactively collect individual data on various platforms to construct giant datasets for enhancing the quality of generative data. However, the training data contains sensitive information about individuals, which is highly susceptible to privacy leakage. A study shows that the larger the amount of training data, the higher the privacy risk will result (Plant et al., 2022). Specifically, in the training, private data can easily be memorized in model weights. In the interaction with users, generative data may replicate the training data, which poses a potential privacy threat. Such data replication is defined as an under-explored failure mode of overfitting, which exists in various generative models (Meehan et al., 2020).

In language generative models, Carlini et al. (Carlini et al., 2021) extracted training data by querying large language models (LLMs). The experiments employ GPT-2 as a demonstration to extract sensitive individual information, e.g., name, email, and phone number. Tirumala et al. (Tirumala et al., 2022) empirically studied the memorization dynamics over language model training, and demonstrated that larger models memorize faster. Nicholas et al. (Carlini et al., 2023b) described three log-linear relationships to quantify the extent to which LLMs memorize training data under different model scales, times of sample replications, and number of tokens. Their experiments indicated that memory in LLMs is more prevalent than previously believed and that memorization scales log-linearly with model size.

In vision generative models, the general view is that generative adversarial networks (GANs) tend not to memorize training data under normal training settings (Webster et al., 2019). However, Feng et al. (Feng et al., 2021) showed experimentally that GANs replication percentage decays exponentially with respect to dataset size and complexity. Stronger memory exists in diffusion models (Webster, 2023; Carlini et al., 2023a; Somepalli et al., 2023). Carlini et al. (Carlini et al., 2023a) illustrated the ability of diffusion models to memorize individual images from the training data and can reproduce them at generation time. Unlike DMs that accept training data as direct input, generators for GANs are trained using only indirect information (i.e., gradients from the discriminator) about the training data. Therefore, GANs are more private. Somepalli et al. (Somepalli et al., 2023) proposed image retrieval frameworks to demonstrate that the generated images by diffusion models are simple combinations of the foreground and background objects of the training dataset. As shown in Fig. 4, diffusion models just create semantically rather than pixel-wise objects identical to original images.

Researchers have suggested some available solutions to mitigate data replication for privacy protection. Bai et al. (Bai et al., 2022) proposed memorization rejection in the training loss, which abandons generative data that are near-duplicates of training data.

Deduplicating training datasets is also a possible option. OpenAI has verified its effectiveness using the distributed nearest neighbor search on Dall-E2. Nikhil et al. (Kandpal et al., 2022) studied a variety of LLMs and showed that the likelihood of a duplicated text sequence appearing is correlated with the number of occurrences of that sequence in the training data. In addition, they also verified that replicated text sequences are greatly reduced when duplicates are eliminated.

Differential privacy (Abadi et al., 2016) is a recommended solution, which introduces noises during training to ensure the generative data is differentially private with respect to the training data. RDP-GAN (Ma et al., 2023b) adds differential noises on the value of the loss function of a discriminator during training, which achieves a differentially private GAN. DPDMs (Dockhorn et al., 2022) enhances privacy via differentially private stochastic gradient descent, which also allows the generative data to retain a high level of availability to support multiple vision tasks. Compared to DPDMs, Ghalebikesabi et al. (Ghalebikesabi et al., 2023) enabled to accurately train larger models and to achieve a high utility on more challenging datasets such as CIFAR-10. PrivImage (Li et al., 2024a) was proposed based on that the distribution of public data used should be similar to the distribution of sensitive data. Therefore, PrivImage elaborately selects similar distribution pre-training data, facilitating the efficient creation of DP datasets with high fidelity and utility. Unlike existing work that only considers DM as a regular deep model, dp-promise (Wang et al., 2024a) was the first work to implement (approximate) DP using DM noise, It employs a two-stage DM training process to reduce the overall noise injection, effectively achieving the privacy-utility tradeoff.

Detecting replicated content is a remedial solution, which detects whether the generative data is in training data and then determines whether to use it. Stability AI provides a tool (Bea, 2022) to support the identification of the replicated images. Somepalli et al. (Somepalli et al., 2023) developed image similarity metrics that are diverse on self-supervised learning and based on an image retrieval framework to search for copying behavior.

Machine unlearning (Bourtoule et al., 2021) can help generative models forget the private training data, which avoids the effort of re-training the model. Kumari et. al (Kumari et al., 2023) fine-tuned diffusion models by modifying the sensitive training data so that the models forget already memorized images. Forget-Me-Not (Zhang et al., 2024c) is adapted as a lightweight model patch for Stable Diffusion. It effectively removes the concept of containing a specific identity and avoids generating any face photo with the identity.

To protect face privacy, many works (Wang et al., 2023d; Hukkelås and Lindseth, 2023; Wen et al., 2023; Yuan et al., 2022; Kim et al., 2023; Liu et al., 2023a; Lyu et al., 2023b; Wang et al., 2024b; Wen et al., 2022) generate a surrogate virtual face with new identity. DeepFace2 (Hukkelås and Lindseth, 2023) generates realistic anonymized images by conditioning GANs to fill in images that obscure facial regions. In order to preserve attributes, Gong et al. (Gong et al., 2020) replaced identity independently by decoupling identity and attribute features, which achieves a trade-off between identity privacy and data utility. To facilitate privacy-preserving face recognition, IVFG (Yuan et al., 2022) generates identifiable virtual faces bound with new identities in the latent space of StyleGAN. In addition, publicly available face datasets for training face recognizers often violate the privacy of real people. For this, DCFace (Kim et al., 2023) creates a generative dataset with virtual faces via diffusion models.

Other works are devoted to generating adversarial faces to evade unauthorized identification. DiffProtect (Liu et al., 2023a) adopts a diffusion autoencoder to generate semantically meaningful perturbations, which can promote the protected face identified as another person. 3DAM-GAN (Lyu et al., 2023b) generates natural adversarial faces by makeup transfer, improving the quality and transferability of generative makeup for identity concealment.

Beyond face, many types of data have sensitive information that needs to be protected (Lu et al., 2023d; Zhang et al., 2022). TrajGen (Cao and Li, 2021) uses GAN and Seq2Seq to simulate the real data to generate mobility data, which can be shared without privacy leakages, thus contributing to the open source process of mobility datasets. In the recommendation systems, UPC-SDG (Liu et al., 2022) can generate virtual interaction data for users according to their privacy preferences, providing privacy guarantees while maximizing the data utility. SinGAN-Seg (Thambawita et al., 2022) uses a single training image to generate synthetic medical images with corresponding masks, which can effectively protect patient privacy when performing medical segmentation. PPUP-GAN (Yao et al., 2023) generates new content of the privacy-related background while maintaining the content of the region of interest in aerial photography, which can protect the privacy of bystanders and maintain the utility of aerial image analysis. Hindistan et al. (Hindistan and Yetkin, 2023) designed a hybrid approach to protect industrial Internet of Things (IIoT) data based on GANs and differential privacy, which causes minimal accuracy loss without extra high computational costs to data processing.

Digital watermarking (Liu et al., 2023d) is a technique used to inject visible or hidden identified information into digital media. The use of digital watermarking in AIGC can achieve a variety of functions:

• •

  Copyright protection: By embedding watermarks with unique identified information, the source and ownership of the data can be traced and proved.

• •

  Authenticity detection: By detecting and identifying the watermark information, it is easy to confirm whether the data is generative and even which models generate it.

• •

  Accountability: It is possible to track and identify the content’s dissemination pipelines and usage, further ensuring accountability.

Depending on whether the watermark is directly produced by the generative model or not, existing works can be categorized into model-specific watermarking and image-specific watermarking, which is shown in Fig. 6.

Yu et al. (Yu et al., 2021) and Zhao et al. (Zhao et al., 2023) implanted watermarking in training data to retrain GANs or DMs from scratch, respectively. Watermarking can also exist in generative data, as they would learn the distribution of the training data. Compared to GANs which directly embed control information into deep features, DMs embed control information multiple times by progressive random denoising, which can improve steganography and stability. Therefore, DMs have the potential to be better in controllability. Stable signature (Fernandez et al., 2023) integrates image watermarking into latent diffusion models. By fine-tuning the latent decoder, the generated data would contain invisible and robust watermarks, i.e., binary signatures, which support after-the-fact detection and identification of the generated data. Cheng et al. (Xiong et al., 2023) introduced a flexible and secure watermarking. The watermark can be altered flexibly by modifying the message matrix, without retraining the model. Additionally, attempts to evade the use of the message matrix result in degraded generated quality, thereby enhancing the security.

Some works (Liu et al., 2023b) can only generate watermarked data when specific triggers are activated. Liu et al. (Liu et al., 2023b) injected watermarking into the prompt of LDMs and proposed two different methods, namely NAIVEWM and FIXEDWM. NAIVEWM activates the watermarking with a watermark-contained prompt. FIXEDWM enhances the stealthiness compared to NAIVEWM, as it can only activate the watermarking when the prompt contains a trigger at a predefined position. PromptCARE (Yao et al., 2024) is a practical a prompt watermark for prompt copyright protection. When unauthorized large models are trained using prompts, copyright owners can input the trigger to verify whether the output contains the specified watermark.

Zeng et al. (Zeng et al., 2023) constructed a universal adversarial watermarking and injected it into an arbitrary pre-trained generative model via fine-tuning. The optimal universal watermarking can be found through adversarial learning against the watermarking detector. Practicably, the secured generative models can share the same watermarking detector, eliminating the need for retraining the detector when it comes to new generators. As the size of the generative model increases, the design of model-specific watermarking will pay more attention to how to use a small number of samples to update a small number of parameters, thereby reducing resource consumption.

Distributed ledger-based blockchain can be used to explore a secure and reliable AIGC-generated content framework.

• •

  Transparency: Blockchain can be used to enable transparent traceability of generative data. Each generative data can be recorded in a block in the blockchain and associated with the corresponding transaction or generation process. This enables users and regulators to understand the source and complete generation path of the generative data.

• •

  Copyright protection: Blockchain can provide a reliable mechanism for copyright protection of generative data. By recording copyright information on the blockchain, it can be ensured that generative data is associated with a specific copyright owner and is available for verification. This can reduce unauthorized use and infringement and provide content creators with evidence of copyright protection.

• •

  Decentralized content distribution: Generative data is stored in a distributed manner across the blockchain network, rather than centrally stored on a single server. This improves the availability and security of generative data and reduces the risk of single points of failure and data loss.

• •

  Rewards and incentives: Through smart contracts, the blockchain can automatically distribute rewards for generative data and ensure a fair and transparent distribution mechanism. This can incentivize contributors to provide higher quality and more valuable generative content.

Du et al. (Liu et al., 2024b) proposed a blockchain-empowered framework to manage the lifecycle of AIGC-generated data. Firstly, a protocol to protect the ownership and copyright of AIGC is proposed, called Proof-of-AIGC, deregistering plagiarized generative data and protecting users’ copyright. Then, they designed an incentive mechanism with one-way incentives and two-way guarantees to ensure the legal and timely execution of AIGC ownership exchange funds between anonymous users. AIGC-Chain(Jiang et al., 2024) carefully records the entire life cycle of AIGC products, providing a transparent and reliable platform for copyright management.

Analyzing distinctive features of generated images is also a viable approach to consider. Interestingly, Wang et al. (Wang et al., 2023a) observed that the image generated by the DMs can be reconstructed by approximating the source model, while the real image cannot. Therefore, they proposed a novel image representation called diffusion reconstruction error (DIRE), which measures the distance between the input images and the reconstructed one. DIRE provides a reliable, simple, and generalized method to differentiate between real images and diffusion-generated images. SeDID (Ma et al., 2023a) leverage the unique properties of diffusion models, namely deterministic inverse and deterministic denoising computational error. In addition, its use of insights from member inference attacks to emphasize distributional differences between real and generative data enhances the understanding of the security and privacy implications of diffusion models. Zhong et al. (Zhong et al., 2023) focused on the inter-pixel correlation contrast between rich and poor texture regions within an image, and presented a universal detector which can generalize to various AI models, including GAN-based and DM-based models. Existing state-of-the-art detectors have a generalization of cross architectures, but the generalization of cross concepts is not considered. To this end, Dogoulis et al. (Dogoulis et al., 2023) proposed a sampling strategy that takes into account the image quality scores of the sampled training data, and can effectively improve the detection performance in the cross-concept setting.

Innovatively, Bi et al. (Bi et al., 2023) explored the invariance of real images, and proposed a method to map real images to a dense subspace in the feature space, while all generative images are projected outside this subspace. In this way, it can effectively address longstanding issues in generative detection, e.g., poor generalization, high training costs, and weak interpretability.

Regarding the model-based methods (Guo et al., 2023; Chen et al., 2023a), a classification model is usually trained using a corpus. Guo et al. (Guo et al., 2023) proposed a text detector for ChatGPT. The detector is based on the RoBERTa model, which is trained by plain answer text and question-answer text pairs respectively. Chen et al. (Chen et al., 2023a) trained two different text classification models using robustly optimized BERT pretraining approach (RoBERTa) and text-to-text Transfer Transformer (T5), respectively, and achieved significant performance on the test dataset with an accuracy of more than 97%.

He et al. (He et al., 2024) extended current detectors to the potential of text attribution to recognize the source model of a given text. The results show that all these detectors have certain attribution capabilities and still have room for improvement. Moreover, model-based detectors can significantly outperform metric-based detectors.

For visual generative data, a lot of attribution works on GANs (Yu et al., 2019; Bui et al., 2022; Yang et al., 2022; Girish et al., 2021; Yang et al., 2023) has been proposed. RepMix (Bui et al., 2022) is a GAN-fingerprinting technique based on representation mixing and a novel loss. It is able to determine from which structure of GAN a given image is generated. POSE (Yang et al., 2023) tackles an important challenge, i.e., open-set model attribution, which can simultaneously attribute images to seen and unseen models. POSE simulates open-set samples that keep the same semantics as closed-set samples but embed distinct traces.

Recent works have begun to focus on DMs. Sha et al. (Sha et al., 2023) constructed a multi-class (instead of binary) classifier to attribute fake images generated by DMs. Experiments showed that attributing fake images to their originating models can be achieved effectively, because different models leave unique fingerprints in their generated images. Lorenz et al. (lor, 2023) designed the multi-local intrinsic dimensionality (multiLID), which is effective in identifying the source diffusion model. Guarnera et al. (Guarnera et al., 2024) developed a novel multi-level hierarchical approach based on ResNet models, which can recognize the specific AI architectures (GANs/DMs). The experimental results demonstrate the effectiveness of the proposed approach, with an average accuracy of more than 97%.

Intriguingly, a new work (Wang et al., 2023b) can attribute generative data to the training data rather than the source model, necessitating the identification of a subset of training images that contribute most significantly to the generated data. In Fig. 8, we can query the generated data in the training set and evaluate their similarity, which can contribute to protecting the copyright of training data rather than models.

Toxicity presents in generative data involves incongruence with human values or bias directed at particular groups, which has the potential to harm societal cohesion and intensify divisions among different groups. Since the training of AIGC models is based on a large amount of unintervened data, toxicity in training data (Birhane et al., 2021) directly leads to the corresponding toxicity in generative data, which covers a variety of harmful topics, e.g., sexuality, hatred (Birhane et al., 2024), politicalization, violentism, and race bias. Some toxic generative examples are shown in Fig. 9.

Some works (Caliskan et al., 2017; Sheng et al., 2019) have found stronger associations between males and occupations in language models, verifying the gender bias in generative data. The investigations revealed that GPT-3 consistently and strongly exhibits biased views against the Muslim community (Abid et al., 2021). Stable Diffusion v1 was trained on the LAION-2B dataset, which contains images described only in English, making generative data biased towards white culture . Likewise, it was observed that DALLA-E displayed unfavorable biases towards minority groups.

Unlike GANs that only use random noise to generate data, DMs can be guided by additional textual prompts, which increases the risk of generating non-compliant content. As a result, researchers primarily focus on the compliance of data generated by DMs. Qu et. al (Qu et al., 2023) provided a comprehensive safety assessment concerning the generation of toxic images, particularly hateful memes from diffusion models. To quantitatively assess the safety of generative images, a safety classifier is developed to identify toxic images based on the predetermined criteria for unsafe content. Their findings indicated that the utilization of harmful prompts resulted in diffusion models producing a significant quantity of toxic images. Additionally, even when prompts are innocuous, the potential for generating toxic images persists. Overall, the danger of large-scale generation of toxic images is imminent.

AI-generated data may be contrary to the facts (Wang et al., 2023c), which is harmful to the public through misleading cognition. For example, ChatGPT may produce responses that sound reasonable and authoritative but are factually incorrect or nonsensical. Even worse, AIGC often explains its generated responses. When AIGC fails to provide accurate responses to the queries, it not only delivers incorrect information but also supplements seemingly plausible explanations. This enhances the inclination of users to place greater trust in these erroneous contents. The United States news credibility assessment and research organization, NewsGuard, conducted a test on ChatGPT (nyt, 2023). Researchers posed questions to ChatGPT containing conspiracy theories and misleading narratives and found that it could adapt information within seconds, generating a substantial amount of persuasive yet unattributed content.

When used in important domains, such unfactual generative data will bring serious harmful impacts (Bender et al., 2021). In the healthcare domain, medical diagnosis requires interpretable and correct information. Once AI-generated diagnostic advice is factually incorrect, it will cause irreparable harm to the patient’s life and health. In the journalism domain, news that distorts the facts will mislead the public and undermine the credibility of the media. In the education domain, the dissemination of incorrect knowledge to students will confuse their minds, thus seriously hampering their academic growth and cognitive development.

Efforts to eliminate toxicity can be divided into four categories. The first one is dataset filtering. A non-toxic training dataset is key to ensuring the security of generative data. Some works (Schuhmann et al., 2022; DAL, 2022; Henderson et al., 2022) have implemented comprehensive processes to filter data contained toxic. OpenAI ensures that any violent or sexual content is removed from DALLA-E2 by carefully filtering (DAL, 2022). Henderson et. al (Henderson et al., 2022) demonstrated how to extract implicit sanitization rules from the Pile of Law, providing researchers with a pathway to develop more sophisticated data filtering mechanisms. However, large-scale dataset filtering also has unexpected side effects on the downstream performance (Nichol et al., 2021).

The second one is generation guidance. Ganguli et al. (Ganguli et al., 2022) identified and attempted to reduce the potentially harmful output of language models in a confrontational manner. They found that the reinforcement learning from human feedback is increasingly difficult to red team as they scale, and a flat trend with scale for the other model types. Brack et al. (Brack et al., 2023) investigated to instruct effectively diffusion models to suppress inappropriate content using the learned knowledge obtained about the world’s ugliness, thus producing safer and more socially responsible content. Similarly, safe latent diffusion (SLD) (Schramowski et al., 2023) extends the generative process via utilizing toxic prompts to guide the safe generation in an opposing direction.

The third one is model fine-tuning. Recently, a new term called the ablation concept or ablation forgetting (Gandikota et al., 2023; Kumari et al., 2023; Heng and Soh, 2024) has brought a novel direction to the elimination of toxic content in generative data. Gandikota et. al (Gandikota et al., 2023) studied the erasure of toxic concepts from diffusion model weights via model fine-tuning. The proposed method utilizes an appropriate style as a teacher to guide the ablation of the toxic concepts, e.g., sexuality and copyright infringement. Selective Amnesia (Heng and Soh, 2024) is a generalized continuous learning framework for concept ablation that applies to different model types and conditional scenarios. It also allows for controlled ablation of concepts that can be specified by the user. However, the ablation concept’s ability to explain the various definitions of toxic concepts remains limited. Inspired with social psychological principles, Xu et al. (Xu et al., 2024b) proposed a novel strategy to motivate LLM to integrate different human perspectives and self-regulate their responses.

Lastly, filtering the generated results is also a viable option. Stable Diffusion includes an after-the-fact safety filter to block toxic images. Unfortunately, the filter easily blocks any generated image that is too close to at least one of the 17 predefined sensitive concepts. Rando et. al (Rando et al., 2022) reverse-engineered this filter and then proposed a manual strategy that enables content not related to sensitive concepts to bypass the filter. In addition, existing toxicity detectors (Markov et al., 2023; Lu et al., 2023c) for real data may be able to be updated to be compatible with generative data. This self-correcting mechanism can significantly reduce toxicity and bias.

In order to constrain the non-factualness caused by AIGC “lying”, Evans et al. (Evans et al., 2021) first identified clear standards for AI truthfulness and explored potential ways to establish them.

A reasonable assessment of content factuality is the critical step toward responsible generative data. Goodrich et al. (Goodrich et al., 2019) constructed relation classifiers and fact extraction models based on Wikipedia and Wikidata, by which the factual accuracy of generated text can be measured. Lee et al. (Lee et al., 2022) proposed a novel training method to enhance factuality by utilizing TOPIC PREFIX for better perception of facts and sentence completion as the training objective, which can significantly decrease the number of counterfactuals. They also study the factual accuracy of LLMs with parameter sizes ranging from 126M to 530B and find that the larger the model, the higher the accuracy. This is because the model has a large enough capacity to learn more generalized knowledge, thus reducing the occurrence of counterfactuals.

Alaa et al. (Alaa et al., 2022) designed a three-dimensional metric capable of characterizing the fidelity, diversity, and generalization of generative data from widely generative models in a wide range of applications. SAPLMA (Azaria and Mitchell, 2023) is a simple but powerful method, which only uses the hidden layer activation of LLM to discriminate the factuality of generated statements.

Interestingly, Du et al. (Du et al., 2023) prompted multiple language models to debate their viewpoints and reasoning processes over multiple rounds, and finally come up with a unitive answer. The results indicate that it enhances mathematical and strategic reasoning in the task while reducing the fallacious answers and illusions that modern models are prone to. CRITIC (Gou et al., 2024) requires LLMs to interact with appropriate tools for feedback learning, such as using a search engine for fact-checking or a code interpreter for debugging. The output of the LLM is modified incrementally by the evaluation results of the feedback by the tools. Chen et al. (Chen et al., 2024a) presented a new alignment framework designed to improve LLMs by converting flawed instruction-response pairs into useful alignment data through mistake analysis. By generating harmful responses and analyzing their own mistakes, the framework can improve the alignment with human values. To prevent LLMs from rambling without knowing the answer, Zhang et al. (Zhang et al., 2024a) taught LLMs the ability to refuse to answer. Specifically, they constructed datasets of refusal perception, and then adapted the model to avoid answering questions that were beyond its parametric knowledge.

The construction of generative datasets facilitates the design and evaluation of countermeasures, helping researchers to identify problems and improve techniques, thus advancing the progress of trustworthy generative data. Researchers have now released diverse datasets for utilization.

For text data, most datasets are domain-specific, e.g., Student Essays (Verma et al., 2023) for essays, TuringBench (Uchendu et al., 2021) for news, GPABenchmark (Liu et al., 2023e) for academic writing, SynSciPass (Rosati, 2022) for scientific text, TweepFake and (Fagni et al., 2021) for tweets. Representatively, HC3 (Guo et al., 2023) is a comprehensive ChatGPT generative text that contains tens of thousands of responses with human experts and covers a wide range of domains such as finance, healthcare, law, etc. The pioneering contributions of HC3 also make it a valuable research resource. More practically, MixSet (Zhang et al., 2024b) is the first mixed-text dataset involving both AIGC and human-generated content, covering a wide range of operations in real-world scenarios and bridging a gap in previous datasets.

For visual data, only some datasets are domain-specific mainly in faces such as IDiff-Face (Boutros et al., 2023) and GANDiffFace (Melzi et al., 2023), and most datasets contain general images such as CIFAKE (Bird and Lotfi, 2024) and AutoSplice (Jia et al., 2023). While, they have different limitations, e.g., targeting only a certain class of images or generators, and containing only a small amount of data. Subsequently, several million-scale datasets are released like GenImage (Zhu et al., 2024) and ArtiFact (Rahman et al., 2023), which have the richness of image content and adopt state-of-the-art generators. To the best of our knowledge, DiffusionDB (Wang et al., 2022a) represents the largest-scale visual generative dataset to date. It comprises 14 million images generated only by Stable Diffusion. This unprecedented scale and diversity offer exciting research opportunities for the study of generative image protection.

The construction of benchmark evaluation provides a standardized baseline, which evaluates the effectiveness and innovativeness of new methods by enabling different methods to be fairly compared under the same testing conditions. Meanwhile, it can ensure the reproducibility of research and clearly identify the shortcomings of existing techniques, thus promoting technological progress. We identified the corresponding benchmarks for different information security properties.

For privacy, PrivLM-Bench (Li et al., 2024b) is a multi-perspective privacy evaluation benchmark that empirically and intuitively quantifies the privacy leakage of LLMs and reveals the neglected privacy of inference data in actual usage. Wu et al. (Wu et al., 2024) provided the first comprehensive privacy assessment of prompts learned via visual prompt learning from the perspectives of attribute inference and membership inference attacks.

For controllability, WaterBench (Tu et al., 2023) is the first comprehensive benchmark for watermarking LLMs, which encompasses 9 tasks and evaluates 4 open-source watermarking technologies. WAVES (An et al., 2024) is a comprehensive watermarking benchmark, which establishes a standardized evaluation protocol consisting of various pressure tests and covers advanced image distortion attacks.

For authenticity, DeepfakeBench (Yan et al., 2023) introduces the first all-encompassing benchmark for detecting deepfakes of generative data, addressing the problem of inconsistent standards and lack of uniformity in this area. Lu et al. (Lu et al., 2023b) comprehensively evaluated techniques used for generative detection and also measured the ability of human vision to discriminate between generative data.

For compliance, GenderCARE (Tang et al., 2024) is a comprehensive framework that includes innovative standards, bias assessments, mitigation techniques, and evaluation metrics for quantifying and alleviating gender bias in LLMs. FELM (chen et al., 2023) is a factual evaluation benchmark for generative texts. Through manual annotation of error types, it alerts users to potential errors and guides the development of more reliable large language models.

Existing privacy protection technologies attempt to remove potentially private content in generative data via machine unlearning or concept forgetting. However, such private content removal is not provable. The parameters of the generative model may still conceal the private content, which can be revealed under specific prompts with backdoors. While differential privacy can provide provable privacy guarantees, it requires retraining the model and sacrificing unacceptable usability. Therefore, it is necessary to explore new provable solutions to guarantee privacy. Exploring information theory-based methods may be a desirable research direction, which can directly limit the amount of retained information after private content is removed. Of course, how to measure the amount of information from generative models at this point is also a challenging problem.

To compare the performance of different privacy-preserving technologies, it is crucial to develop quantitative evaluation metrics to measure privacy and utility before and after private content removal. However, constructing such fine-grained evaluation metrics requires systematically analyzing the learning processes and memory mechanisms of different generative models, which is difficult. In addition, compared to text data, the amount of information in visual data is multiple and complex (a picture is worth a thousand words) thus making it more difficult to measure privacy and utility. Considering that visual data is more concerned with semantic-level information, in the future, it is meaningful to design the metrics oriented to the semantic level, which can effectively guide generative data to achieve a balance between utility and privacy. In addition, since the semantics represented by privacy and utility are not always the same in different scenarios, it will be more practical and challenging to make quantitative metrics generic or flexible.

The privacy onion effect refers to the fact that when the most privacy-vulnerable outlier layer is removed, the privacy risk of the previously secure layer becomes the greatest. The existence of this effect can have a variety of serious consequences, in particular, it indicates that privacy-enhancing technologies may actually compromise the privacy of other users. However, there are very few existing exploration studies targeting this effect. The reason that this effect brings about privacy leakage originates from the idea that mean data points are rarely compromised, but outlier samples are often memorized. Therefore, it may be an intuitive solution to consider clustering the dataset to eliminate outlier points, but this perhaps reduces the diversity of the data. In this way, it is an important problem to maintain dataset utility while deleting reasonable data.

Existing work simply adds adversarial perturbations to block access to the expected generative data. However, in most cases, users just want to prohibit generative data from containing specific semantics while allowing other semantics to exist. For example, an image containing the face of a celebrity is accessible, but when it is changed in gender or uglified it is not accessible. Existing work prohibits generative data with all semantics, which limits its practicality. Furthermore, users with roles can be restricted from accessing specific data, such as teenagers being prohibited from accessing essay writing type data but allowing adults to access it. Therefore, it is also a promising direction to provide fine-grained access control for generative data, which allows for flexible usage. To achieve it, a deep understanding of its generative process and digging into its intrinsic mechanism are necessary.

Unlike visual data which is easy to achieve robust watermarking with high capacity, text data faces important challenges: Firstly, text data is significantly sparse, for example, the maximum token limit in GPT-4 is 8.2k which is much smaller than the watermarking capacity that can be embedded in 512-pixel images. Secondly, the semantics of text data are fragile as subtle changes may confuse or compromise its semantics, whereas minor changes in images can maintain consistent semantics. Some multi-bit watermarking can increase the watermarking capacity to some extent, but it also significantly reduces the robustness. Therefore, in future research, it is essential to consider designing new text watermarking paradigms to effectively balance the high capacity and strong robustness of text watermarking.

Existing detection methods attempt to find the decision boundary between real and generative data by a specific generative model, but it is difficult to generalize to new generative models. Detection performance degrades significantly when encountering data generated by generative models not seen in the training set. To improve generalization, the detector needs to be fine-tuned to allow capturing the decision boundaries of new generative models. However, the decision boundaries captured by the detector are intricate as the generative model iteratively changes. Since real data does not change massively over time, its inherent invariances (Qi et al., 2024) make it a reliable clue for generative detection. These invariances may include signal distribution, noise distribution, texture features of the data, etc. Exploring new invariants or joining multiple invariants is an important future direction. Meanwhile, interpretable analysis of these invariants will also make generative detection more credible and thus be able to be used for judicial forensics.

Similar to the real data, with the widespread use of generative data, it is becoming increasingly important to accurately detect whether the generative data has been tampered. However, current existing generative detection methods are challenging to do. The main reason is that generative data is itself generated by AI and also tampered with in an AI manner, making it difficult to judge modifications based on the fingerprints of the AI models. To address this issue, one can consider embedding semi-fragile watermarks for generative data, thus ensuring its integrity. However, it is unrealistic for all users to uniformly adopt such post-hoc operations. Therefore, there is still a requirement to propose effective techniques to detect the tampered generative data directly. To do this, researchers need to conduct a variability analysis of AI-generated models and AI-tampered models.

When it comes to important decisions, machine vision-based generative detection techniques only provide initial judgments, while users still need to make further judgments by themselves. However, existing technology does not yet provide a directly interpretable, trustworthy, and easy-to-handle detection modality which makes it difficult for users to make rational decisions conveniently. Therefore, the development of interpretable, trustworthy, and easy-to-handle detection tools will be able to attract more social attention and popularization, thus promoting the friendly usage of detection techniques for disadvantaged groups, including unskilled users and the elderly. Conducting relevant user research analysis and incorporating explainable AI will help build user-friendly generative detection.

While existing laws and regulations already impose requirements for compliance with generative data, how to quantify and assess these requirements remains a challenging question. Firstly, compliance requirements are abstract and broad. Laws and regulations usually stipulate a series of principles and standards, but do not give specific metrics and standards. Therefore, it requires an in-depth understanding of the connotations of the legal provisions and their concretization in the context of the actual situation. Secondly, the type of generative data is complex, which covers a variety of multimodal and cross-modal contents. The automated methods for quantification involve the cross-application of fields, which requires the development of efficient multimodal fusion algorithms to realize it. Finally, there are differences in laws and regulations in different countries and regions. Compliance assessment of generative data needs to consider and cater to different geographical and cultural contexts, which adds to the complexity of the assessment.

Toxic content hiding in generative data often remains imperceptible to human perception and toxicity detectors. However, it might be uncovered by the special extractor of attackers, thereby giving rise to potential hazards. This situation may stem from the deep embedding of toxic information or the utilization of advanced information-hiding techniques to evade conventional detectors. Such hidden toxic content carries an undeniable risk that could disseminate widely across platforms, e.g., social media, news reporting, and virtual communities, consequently precipitating a range of societal issues. To counter this, human intelligence has advanced reasoning capabilities and is able to detect potentially toxic content by recognizing language, context, and metaphor. Of course, relying solely on human intelligence is extremely time-consuming and thus impractical. Future work should focus on human-intelligence-guided toxicity detection, which aims to find more stealthy toxicity using less human labor.