GMValuator: Similarity-based Data Valuation for Generative Models

Let us denote $X=\left\{x_{1},...,x_{n}\right|x\sim\mathcal{X}\}$ as the training dataset without duplicated data for a generation task, and $n$ is regarded as the size of the dataset. Denote the generated dataset as $\hat{X}=\{\hat{x}_{1},...,\hat{x}_{m}\}$ by a well trained generative model $G^{*}$ (e.g., Generative Adversarial Network (GAN), Variational Auto-encoder (VAE), Diffusion Model) on $X$.

The core of these generative models is an intractable probability distribution learning process to train a generator $G$, that maps each $z$ sampled from the noise distribution $\mathcal{Z}$ to a generated sample $\hat{x}$ in estimated distribution by maximizing the likelihood $p_{\mathcal{X}}(x)\approx\int p_{G}(x|z)p_{\mathcal{Z}}(z)dz$. This can be achieved by minimizing the distance $d(\cdot,\cdot)$ (i.e., , Wasserstein Distance) between the estimated distribution and training data distribution:

Therefore, the data generated by an optimal generator, which closely approximates the training distribution, can be considered as drawing from a subdistribution of $\mathcal{X}$.

The primary objective of our research is to tackle the issue of data valuation in generative models using black-box access. In essence, given a fixed set of data samples $\hat{X}$ with a size of $m$ generated from the well-trained deep generative model $G^{*}$, our aim is to determine the value $\phi_{i}(x_{i},\hat{X},G^{*})$ associated with each data point $x_{i}\in X$ for $i\in[n]$ in the deduplicated training dataset, which contributes to the generated dataset. We denote the value for training data $x_{i}$ as $\phi_{i}$ in the rest of the paper for simplicity.

The high-value data will achieve a smaller distance to the target distributions, thus, a better approximation. Therefore, data valuation in the above problem formulation contains two steps: Step 1, calculating all of the score value $\mathcal{V}(x_{i},\hat{x}_{j})$; Step 2, mapping the contribution from training data to generative data based on the scores $\mathcal{V}$.

However, there are several open questions and challenges in performing the above two steps for calculating Eq. equation 5:
Challenge 1: Efficiency. In step 1, considering $n$ training samples and $m$ generated samples, where $\mathcal{O}(C)$ represents the complexity of the selected pair-wise distance calculation, the total complexity of this step amounts to $\mathcal{O}(mnC)$. In practical scenarios with large training datasets (e.g., $n>10K$), the computation cost becomes prohibitively expensive. Additionally, fitting such a large collection of high-dimensional data for distance calculation can pose significant challenges in system memory.
Challenge 2: Contribution plausibility. To ensure that a training data point contributes more if it is similar to high-quality generated data and less if it is similar to low-quality generated data, the contribution scores should be adjusted based on the quality of the generated data.
Challenge 3: Non-zero scores. In practical scenarios, the distance between training data and the least similar generated data is not infinite, which may result in false non-zero contribution scores. With a large dataset size, the accumulation of these noisy scores yields biased data valuation.

In light of this, we present a novel and efficient data valuation approach suitable for agnostic generative models, termed as GMValuator as elaborated in Section 3.

Considering the complexity of calculating $\mathcal{V}(x_{i},\hat{x}_{j})$, we formulate it as an ESM problem between generative data and training data. Each generated data sample is matched to several training data samples based on their similarity. We denote $\mathcal{P}_{j}=\left\{x_{1},x_{2},...,x_{k}\right\}=f(X,\hat{x}_{j})$ as the subset of training data that contains the $k\ll n$ most similar data samples. Here, $f$ represents the similarity-matching strategy, encompassing the recall and re-ranking phases, which will be introduced subsequently.

Recall Phase: The main aim of the recall phase is to rapidly identify a subset of training samples that are similar to a generated sample. To achieve this, an initial step involves encoding all original training images and generated samples from the image space $R^{C}$ to a lower-dimensional embedding space $R^{D}$ using an pre-trained encoder $f_{e}$ (such as CLIP Radford et al. (2021)) to reduce computational complexity. Subsequently, the technique of Product Quantization (PQ)  Jegou et al. (2010) is employed to further decrease the computational burden. Specifically, PQ divides embedding vectors into subvectors and independently quantizes each subvector through $Q$-means clustering. This process generates compact PQ codes that serve as representations of the original vectors. This representation significantly reduces the vector sizes, allowing for efficient estimation of Euclidean Distance between two samples. By incorporating the recall method into the similarity matching process, the computational complexity is lowered from $\mathcal{O}(mnC)$ to $\mathcal{O}(mQD)$, where $D\ll C$ and $Q\ll n$. Consequently, generated images can quickly identify their top-$k$ most similar training data samples.

Re-Ranking Phase: Following the PQ-based efficient recall process in GMValuator, we further improve the precision of the results by utilizing perceptual similarity Fu et al. (2023) for precision ranking. Once we have extracted perceptual features from the top $k$ recalled training samples, we proceed to calculate the distance for each pair of items. To obtain precise distance measurements based on their perceptual content, we propose to use Learned Perceptual Image Patch Similarity (LPIPS) Zhang et al. (2018) or DreamSim Fu et al. (2023) as the distance measurement $d$ to gain insights into the perceptual dissimilarity between the generated sample and different training samples. These metrics enable us to precisely measure the most significant contributors according to their perceived similarity, more importantly, the obtained distance $d$ will be employed to compute data valuation in Eq. equation 6. We do not use Wasserstein distance as it is more proper to measure dissimilarity between two probability distributions rather than a pair of image instances with semantic characteristics.

To tackle challenge 2, we have to establish a connection between the quality of the generated sample that the training samples contribute to and the contribution score. This is necessary before assigning contribution scores to the ranked training samples for the generated sample. For low-quality generated samples $\hat{x}_{\rm low}$, we expect their total contribution score from contributors to be lower compared to high-quality samples $\hat{x}_{\rm high}$, namely $\sum_{i\in[k]}\mathcal{V}(x_{i},\hat{x}_{\rm low})<\sum_{i\in[k]}\mathcal{V}(x_{i},\hat{x}_{\rm high})$. This motivation arises from the observations in Figure 3, where the first and second rows show generated samples using $z$ from a normal distribution and uniform distribution, respectively. The values denoted in Figure 3 directly employs Eq equation 4, so we have $\sum_{x_{i}\in\mathcal{P}}\mathcal{V}(x_{i},\hat{x})=1$ for $\forall\hat{x}\in\hat{X}$. The real samples are noticeably dissimilar to the generated data, yet they are still assigned a high value.

Therefore, we propose to calibrate the contribution scores with the generated data quality. Specifically, we obtain a comprehensive quality score $q_{j}\in[0,1]$ for each generated image integrated using MANIQA Yang et al. (2022) model into our evaluation process. A higher $q_{j}$ indicates better data quality. This score of MANIQA considers various factors such as sharpness, color accuracy, composition, and overall visual appeal. We also provide the necessity of this calibration in the appendix I. Incorporating the image quality evaluation provided by MANIQA allows us to more accurately assess the generated images and take into account their perceptual fidelity and aesthetic qualities.

Finally, we utilize the image quality $q$ to calibrate the contribution scores. Notably, during the recall phase of similarity matching, we select the top $k$ contributors $\mathcal{P}_{j}$ for the generated data $\hat{x}_{j}$. Consequently, we only consider the contribution score between $\mathcal{P}_{j}$ and $\hat{x}_{j}$ by setting $\mathcal{V}(x_{i},\hat{x}_{j})=0$ for $x_{i}\notin\mathcal{P}_{j}$. This strategy addresses challenge 3 by assigning zero scores to irrelevant samples, effectively reducing bias and noise in value estimation when dealing with a large $n$. Hence, we define the contribution score of each training data point for a specific synthetic data point as follows:

which is used to give different scores to training data points according to their ranking distance. The final data value is obtained by plugging Eq. 6 into Eq. 5.

Datasets. Our experiment settings are listed in Table 1. Particularly, we consider different types of generative models, including GAN based models, $\beta$-VAE Higgins et al. (2016), and diffusion models Ho & Salimans (2022); Wang et al. (2022). The generation tasks are conducted on benchmark datasets (i.e., MNIST LeCun et al. (1998) and CIFAR Krizhevsky et al. (2009)), face recognition dataset (i.e., CelebA Liu et al. (2018)), high-resolution image dataset with size 512 $\times$ 512, and 1024 $\times$ 1024 (i.e., AFHQ Choi et al. (2020), FFHQ Karras et al. (2019)), the large-scale dataset with 1,000 classes and 14,197,122 images (i.e., ImageNet Deng et al. (2009)), and text-to-image dataset (i.e., Naruto Cervenka (2022)).

Baselines. Note that GMValuator is a model-agnostic method. To showcase its efficacy, we compare it with two baseline methods: VAE-TracIn Kong & Chaudhuri (2021) and IF4GAN Terashita et al. (2021). VAE-TracIn identifies the most significant contributors for a generated sample, which is the same as the similarity matching process of GMValuator (Sec. 3.1). Thus, we compare GMValuator with VAE-TracIn in Sec. 4.3 and Sec. 4.4. IF4GAN finds the high-valued training samples for generative model training, which we regard it as the baseline method in section 4.5. We also examine our approach using DreamSim, LPIPS and $l_{2}$-distance in the re-ranking phase, referred to as GMValuator (DreamSim), GMValuator (LPIPS) and GMValuator ($l_{2}$-distance) respectively, while the GMValuator without the re-ranking phase is referred to as GMValuator (No-Rerank).

Embedding Appoarch. In this experiment, we utilize CLIP as an example choice for a pre-trained encoder $f_{e}$ in the recall phase of efficient similarity matching. To demonstrate the generalizability of GMValuator, we also do experiments using more embedding approaches in Sec. G of the appendix.

Given the absence of a definitive benchmark to evaluate data valuation methods in the context of generative models, we propose to evaluate the truthfulness of data valuation outcomes through the examination of four criteria (C):
- C1: Identical Class Test. Following the concept of an identical class Hanawa et al. (2020), we posit that the most significant contributors among the training samples should belong to the same class as the generated sample produced by a well-trained generative model, which is also used for evaluation in VAE-TrancIn Kong & Chaudhuri (2021).
- C2: Identical Attributes Test. Following C1, we extend the concept of identical class test and consider the attributes of samples as the ground truth to examine the most significant contributors by our approach on datasets without class labels. We examine the overlap level of attributes (i.e., the number of identical attributes) between the most significant training samples with the generated sample.
- C3: Out of Distribution Detection. Under the assumption, the out of distribution training data samples (e.g., noisy samples) can be identified as low-contribution training samples for the generated dataset. To evaluate the performance of data valuation approaches, we consider the contribution level of noisy data samples as the performance metric across various approaches.
- C4: Efficiency. As an efficient training-free approach, GMValuator should measure the data value in a limited time and be much more efficient than the previous work while obtaining truthful results.

Methodology. In this subsection, we follow setup of VAE-TracIn Kong & Chaudhuri (2021) by training separate $\beta$-VAE models on MNIST Krizhevsky et al. (2009), CIFAR LeCun et al. (1998). We then attribute the most significant contributors by VAE-TracIn and our approach over generated samples. By the concept of identical class test, we expect a perfect data valuation approach should indicate training data in the same subclass of the generated sample $\hat{x}_{j}$ contribute more to $\hat{x}_{j}$. We examine GMValuator (DreamSim), GMValuator (LPIPS), GMValuator ($l_{2}$-distance) and GMValuator (No-Rerank) separately. In addition to using VAE and comparing it with the baseline method, we also perform experiments using masked diffusion transformer on the large-scale dataset (ImageNet) and other various generative models (in the appendix) to demonstrate the model-agnostic property of GMValuator.

Results. For a given generated data, we examine the class(es) of is top $k$ contributors in the training data. We count the number of training $Q$ samples in the top $k$ contributors, which have the identical class as the generated data. The identical class ratio $\rho=Q/k$. We report the averaged $\rho$ over the generated datasets (the data size $m$=100) on different choices of $k$ in Table  2. GMValuator has the most contributors that belong to the same class with the generated sample among on MNIST and CIFAR-10, while GMValuator (DreamSim) has the best performance. The results for CIFAR-10 using VAE-TracIn are extremely bad. This could be attributed to underfitting, as the authors mentioned in their study Kong & Chaudhuri (2021), training a good $\beta$-VAE model on CIFAR-10 is challenging. In addition, GMValuator (DreamSim) shows a significant advantage on ImageNet since it considers semantic characteristics while $l_{2}$-distance or LPIPS destroy the performance, while the experiments conducted on ImageNet using baseline method (VAE-TracIn) is impractical due to its unacceptable computational overhead. In the appendix, further analysis and explanation for this phenomenon are discussed and the results from various generative models are presented as well.

Methodology. In extension to C1, we focus on certain image attributes instead of class labels as the ground truth. We posit that the most significant contributors to a generated sample should share similar attributes with it. We train Diffusion-StyleGAN Wang et al. (2022) on CelebA, and leverage our approach to identify the most significant contributors in training samples and use some attributes including hat, gender, and eyeglasses as the ground truth, to check the correctness of our approach for identifying the most significant contributors for a generated sample. We also examine this on AFHQ, FFHQ dataset by StyleGAN Choi et al. (2020), and text-to-image dataset Naruto Cervenka (2022) by stable diffusion model Rombach et al. (2022).

Results. The results in Table 3 indicate that GMValuator can find the most significant contributors with the same attributes to the generated sample. Besides, using DreamSim in the similarity re-ranking phase obtains the best performance over these three attributes. We also visualize some visible results in Figure 5 on CelebA, and AFHQ, FFHQ in the appendix. As we can see, the most significant contributors have similar attributes (e.g., skin color, hair) to the generated sample. Similarly, the visualized results on AFHQ and FFHQ shown in the appendix demonstrate that the top $k$ contributors have similar attributes to the generated sample such as the fur color of cats or dogs in AFHQ or hair color in FFHQ. In addition, the results on the text-to-image dataset Naruto Cervenka (2022) corroborate the same conclusion, as shown in the appendix N.

Methodology In this experiment, we introduce 100 noisy images into the MNIST dataset to create a new corrupted training dataset and each noisy image is generated by adding Gaussian noise to the clean data. These contaminated samples can diminish the performance of generative models, and as a result, they are anticipated to possess lower values compared to the rest of the training dataset. Our objective is to assess the performance of both GMValuator and its baseline method, IF4GAN Terashita et al. (2021), by analyzing the value rankings of noisy samples. An effective approach should place noisy data in lower positions within the value ranking. We conducted this evaluation using 10,000 generated images and referred to the study Terashita et al. (2021) for the baseline method (IF4GAN).

Results. The results depicted in Figure 5 demonstrate that the values of all noisy training samples, as calculated by GMValuator, are lower compared to the values calculated by IF4GAN. This observation suggests that the performance of GMValuator is significantly better than that of IF4GAN. Besides, the performance of GMValuator with the re-ranking phase is better than GMValuator (No-rerank).

Methodology. We thoroughly evaluate and compare the efficiency of our approach against existing data valuation methods for generative models discussed above: VAE-TracIn and IF4GAN. Specifically, we measure the attribute time for one generated sample on an average of 100 test samples using both VAE-TracIn and our proposed approach on MNIST and CIFAR-10. Furthermore, under the same setting in C3, we assess the data valuation time when utilizing IF4GAN and our approaches on noise MNIST which we mentioned in C3. In addition, we analyze our computational efficiency as the data scale grows in the appendix O.

Results. The average time taken to attribute the most significant contributors for one generated sample is calculated and compared with the baseline method (VAE-TracIn), as demonstrated in Table 4. Our approaches are all greatly more efficient than the baseline methods on both datasets. When it comes to data valuation time for GAN on C3, GMValuator (No-rerank) and GMValuator (LPIPS) are significantly better than IF4GAN. The aforementioned phenomenon is attributed to the costly nature of the Hessian estimation process, despite the utilization of certain acceleration methods. It is noticeable that GMValuator (DreamSim) is lightly time-consuming when compared to IF4GAN. This lead to a trade-off problem due to the GMValuator using DreamSim in the re-ranking phase obtained the better performance from C1 to C3.

Overall, through the above experiments, GMValuator outperforms than baseline methods, while GMValuator (DreamSim) obtains the best performance. This is due to the fact that DreamSim captures mid-level similarities in image semantic content and layout compared to LPIPS. And the selection of different GMValuators can be combined with the consideration of their efficiency. Furthermore, the results shown in the appendix O also demonstrate that our method maintains consistent computational efficiency even as the data scale increases and presents its scalability on large-scale datasets.