A CLIP-Powered Framework for Robust and Generalizable Data Selection

To alleviate the domain shifts and discrepancies between the pretrained and target datasets, we incorporate dimension-preserving adapters to perform adaptation on the target dataset. The image and text adapters are denoted as $A_{I}$ and $A_{T}$, respectively. Both adapters are fine-tuned for knowledge transfer while the pretrained CLIP weights are frozen. To maintain high efficiency, both adapters utilize simple MLP.

Specifically, the fine-tuning process employs the InfoNCE loss Parulekar et al. (2023); Oord et al. (2018), which maximizes the mutual information between the image and text representations. The text representation describes the category information using the prompt: “A photo of $[\text{CLS}]$”, where the token $[\text{CLS}]$ represents the corresponding category. The loss ensures that the adapters effectively align and capture the relevant features from both modalities. Furthermore, it enhances the model’s ability to distinguish between positive and negative pairs, thereby improving the robustness and accuracy of the deep representations for the specific dataset.

For classification datasets, the learning process for training samples is intrinsically linked to acquiring knowledge of the corresponding categories. A training sample that more accurately represents its category is typically more effective in training deep networks. In this way, the Semantic Alignment Score (SAS) is designed to assess the semantic similarity between training samples and their corresponding categories. Specifically, since image and text features reside within the same embedding space Radford et al. (2021), the SAS is derived by calculating the cosine similarity between the embedded image and corresponding textual deep descriptions. Accordingly, the SAS for the $i$-th sample is defined as:

$$\bm{S}_{Ai}=\mathrm{cos}(A_{I}(E_{I}(\bm{I}_{i})),A_{T}(E_{T}(\bm{T}_{i}))),$$

(1)

where $\bm{I}_{i}$ is the $i$-th sample, $\bm{T}_{i}$ is the textual description of the corresponding category for $\bm{I}_{i}$, $E_{I}$ and $E_{T}$ are pretrained image and text encoders, respectively.

The effects of SAS and SDS are depicted in Figure 3. SDS contributes to the diversity of samples, but the selected data points may contain noise (Figure 3(b)). On the other hand, SAS can select samples that are semantically appropriate, as these points are basically around the sample center (Figure 3(c)). However, these samples may be too concentrated and thus lack diversity. When using SDS and SAS together (Figure 3(d)), we can cover the entire category space with fewer data and select samples that are both semantically representative and diverse, thereby boosting the effectiveness of data selection.

Instead of relying on combinatorial optimization functions for sample selection Killamsetty et al. (2021b), our method determines the selection through SGD multi-objective optimization, which improves computational efficiency and accelerates convergence. Specifically, we introduce a sample-wise parameter $\bm{d}$ to denote the selection decision, where elements of 1 indicate the selection while 0 indicates otherwise. Although binary parameters are difficult to optimize in neural networks due to the absence of gradient, we employ the $\mathrm{sigmoid}(\cdot)$ function to push the continuous values of $\bm{d}$ towards approximate binarization. After optimization, $\bm{d}$ is strictly binarized to explicitly indicate the final sample selection. Initially, $\bm{d}$ is initialized with all 1s.

To guide the optimization process, we introduce three loss items. The first item, $\mathcal{L}_{sa}$, is designed to prioritize samples with high SAS since these samples are more representative of their corresponding categories, which is defined as follows:

$$\mathcal{L}_{sa}=-\frac{1}{N}\sum_{i}^{N}\mathrm{sigmoid}(\bm{d})*\bm{S}_{Ai},$$

(3)

where $N$ is the total number of samples. $\mathcal{L}_{sa}$ punishes samples with low SAS and encourages the selection of samples with better semantic alignment.

In addition, we introduce another loss term, $\mathcal{L}_{sd}$, to encourage the selection of more diverse samples characterized by higher SDS, which is defined as:

$$\mathcal{L}_{sd}=-\frac{1}{N}\sum\mathrm{sigmoid}(\bm{d})*\bm{S}_{Di}.$$

(4)

To mitigate the group effect, we optimize the selected datasets w.r.t. specific selection ratios, aiming to identify the optimal subsets. We introduce a selection loss term, $\mathcal{L}_{s}$, to ensure the selection process adheres to the target ratio. However, deriving exact selection rates from continuous real-valued parameter optimization is difficult. While strictly binarized values facilitate explicit sample selection, they are challenging to optimize through gradient backpropagation. To address this, we utlize the straight-through estimator (STE) Bengio et al. (2013) to estimate the actual selection rate and derive gradients. STE allows gradients to pass through the discrete decisions during backpropagation, effectively combining the benefits of both continuous and binary parameters for efficient optimization and accurate sample selection. In this way, $\mathcal{L}_{s}$ is defined as:

$$\mathcal{L}_{s}=\sqrt{\left[\frac{1}{N}\sum_{i}\mathrm{STE}\left[\mathbb{I}\left(\mathrm{sigmoid}(\bm{d}_{i})_{\mathrm{i}}>0.5\right)\right]-s_{r}\right]^{2}},$$

(5)

where $\mathbb{I}$ is an indicator function, and $s_{r}$ denotes the expected selection ratio. The loss term guides the parameter $\bm{d}$ toward near-binary values, ensuring the count of ones aligns with the expected sample size. Since the selection is guided by adaptive optimization, the final selection ratio may deviate slightly from the target. To minimize this deviation, we constrain $\mathcal{L}_{s}$ with a threshold $\theta$, which in our work is set to $5\times 10^{-4}$, ensuring the actual selection ratio differs by less than $\pm 0.05\%$ from the expected value. Finally, the overall loss function is formulated as:

$$\mathcal{L}=\mathcal{L}_{sa}+\alpha\mathcal{L}_{sd}+\beta\mathcal{L}_{s},$$

(6)

where $\alpha$ and $\beta$ are coefficients that adjust for numerical differences among the loss terms and can be set conveniently.

Datasets and network architectures. Consistent with previous works Tan et al. (2024); Xia et al. (2023); Zheng et al. (2023), we evaluate the effectiveness of our proposed method on various popularly used benchmark datasets, including CIFAR-10/100 Krizhevsky et al. (2009), Tiny-ImageNet Chrabaszcz et al. (2017), and ImageNet-1k Deng et al. (2009). To evaluate the generalization performance of our selected datasets, we study the effectiveness of our proposed method on a wide range of network architectures, including ResNet-18/50 He et al. (2016), Vision Transformer (ViT) Dosovitskiy et al. (2020), Swin-Transformer Liu et al. (2021), VGG-16 Simonyan & Zisserman (2014), and DenseNet-121 Huang et al. (2017).

Baselines. We compare our proposed method with ten most representative SOTA baselines, i.e., (1) Random, (2) MoSo Tan et al. (2024), (3) Glister Killamsetty et al. (2021b), (4) Herding Welling (2009), (5) Forgetting Toneva et al. (2018), (6) GraNd and (7) EL2N Paul et al. (2021), (8) Self-sup.-selection (SSP) Sorscher et al. (2022), (9) CG-Score Nohyun et al. (2023), and (10) Moderate-DS Xia et al. (2023).

Parameter settings. The parameters in our proposed method can be easily set. The coefficient $\alpha$ is proportional to the expected selection rate $s_{r}$, balancing the importance of dataset diversity. For instance, $\alpha$ can be set equivalent to $s_{r}$. The coefficient $\beta$ is set to 2 on all other datasets to adjust the numerical differences among loss items.

Performance Comparisons Consistent with prior works Xia et al. (2023); Sorscher et al. (2022), we report top-1 accuracy on CIFAR-100 and Tiny-ImageNet, and top-5 accuracy on ImageNet-1k. Note that the methods Glister and CG-Score are not compared on ImageNet-1k due to the heavy computation costs. Specifically, Glister obtains iterative solving of the bi-level optimization problem Killamsetty et al. (2021b), while CG-Score involves the calculation of large Gram matrix inversions, making them impractical for large-scale datasets.

As illustrated in Figure 4, our method consistently achieves the best accuracy across all datasets. Particularly, on the more challenging Tiny-ImageNet and ImageNet-1k datasets, our approach outperforms other methods by a notable margin. While existing approaches yield relatively marginal accuracy improvements on the small-scale CIFAR-100 dataset, the gains brought by our method are more substantial. Additionally, with relatively high selection ratios on these datasets, such as 90%, our selected datasets exhibit nearly lossless or even higher performance compared with the original full datasets and other baselines. The results indicate that our method can not only reduce training costs but may also serve as a way for data cleaning.

Generalization Comparisons on Different Architectures  In this section, we evaluate the generalization effectiveness of our selected datasets on deep architectures different from those used in the selection process. Specifically, we employ the VGG-16 and DenseNet-121 models to train on the selected datasets from Tiny-ImageNet. As shown in Table 1, the results indicate that our method surpasses all baseline methods on both architectures, demonstrating its desirable architectural generalization ability. This suggests that the selected datasets are broadly applicable, irrespective of the specific network architecture.

Training Efficiency Comparisons  To evaluate the selection efficiency of various methods, we present an analysis of the balance between effectiveness and efficiency. As shown in Figure 7, our method presents the best performance with desirable efficiency. Herding, EL2N, and GraNd obtain the lowest selection costs because they rely on predefined metrics or select samples in very early training. Our method is slightly slower than them but exhibits higher accuracy. Compared with the optimization-based approaches, like MoSo and Glister, our method enjoys both lower costs and better performance. The results verify the effectiveness of our method in balancing selection efficiency and accuracy.

Robustness on Noisy Labels  Real-world datasets often involve label noise, where some sample labels are incorrectly flipped, resulting in mislabeled data. Unfortunately, creating clean and diverse datasets is time-consuming and expensive. Therefore, it is necessary to evaluate the performance of selection methods under such complex scenarios. In this study, we introduce symmetric noises Li et al. (2022) to generate mislabeled data on both C-100 and T-ImageNet, with a 20% noise rate.

As can be seen in Table 4.2, our approach exhibits superior robustness to noisy labels, significantly outperforming other baselines by a large margin. Specifically, our approach yields improvements of over 10.12% on CIFAR-100 and 4.41% on Tiny-ImageNet compared to previous leading methods. Additionally, Table 4.2 shows the distribution of noisy data within the selected datasets, where our method selects only 0.24% noisy samples, considerably fewer than other baselines. This reduction in noise underscores the potential of our method to improve overall data quality.

We argue that the robustness of our approach can be attributed to $\bm{S}_{A}$ in Eq. 1, which assesses the semantic alignment between image content and its labels. In the presence of label noise, this alignment is disrupted, resulting in a lower SAS, which in turn reduces the likelihood of such samples being selected during optimization. In contrast, most baseline methods rely solely on image features for selection, which may result in performance degradation when faced with noisy labels. In some cases, the performance of these methods is even worse than that of random selection. While Moderate also selects a relatively low proportion of noisy data, its performance is worse than ours. This discrepancy highlights the effectiveness of our method in making more strategic selections in noisy environments, thereby not only minimizing noises but also optimizing the selected datasets for improved generalization.

Robustness on Corrupted Images  We further evaluate the performance of our proposed method on real-world noise corruptions that are frequently encountered Singh et al. (2024). To simulate such corruptions, we employ the following five types of realistic noises Xia et al. (2023), including Gaussian noise, random occlusion, resolution, fog, and motion blur. The corruption rate is set to 5%, 10%, and 20%, respectively.

As shown in Figure 5, compared with prior baselines, our approach consistently presents greater robustness to corrupted images across varying corruption rates, demonstrating strong generalization in these challenging scenarios. Notably, even at a high corruption rate of 20%, our method maintains desirable generalization performance. This robustness is primarily attributed to the integration of text modality into the selection process, alongside the image modality. The SAS defined in Eq. 1 measures the alignment between the image features and their corresponding category features. When images are corrupted, this alignment is disrupted, thereby reducing the SAS and correspondingly decreasing the likelihood of selecting those images. In contrast, methods such as Forgetting tend to prioritize difficult training samples, potentially making corrupted images more likely to be selected, as these images are typically more difficult to correctly classify. As a result, these methods are less robust to corrupted images, leading to a deterioration in generalization performance.

Visualization Analysis  To demonstrate the generalization of the selected datasets, we train two models using the original dataset and the selected dataset (90% selection ratio), respectively, and obtain their embedding results on the CIFAR-10 test set. To visualize the dataset distribution, we apply t-SNE to the embeddings generated by both models. The visualization in Figure 7 shows that the model trained on the selected dataset produces better embedding results: a better inter-cluster separation and intra-cluster compactness. For a quantitative analysis, we use the Dunn Index (DI) Ncir et al. (2021) to evaluate the clustering results (the higher, the better). After removing 10% of the data, the DI increases by 43%, presenting better clustering results.

Generalization to More Advanced Architectures We further employ the selected datasets to train more advanced ViT-based architectures, including Swin Transformer, ViT-Base, and ViT-Large. From Table 4, and corroborated by the results from previous sections, our selected datasets consistently achieve lossless performance across both CNN-based and Transformer-based architectures with reduced training costs. This demonstrates that our approach obtains highly generalizable datasets applicable to a wide range of network architectures.

Generalization to More Challenging Benchmark Datasets To further evaluate the generalization and robustness of models trained on our selected datasets, we conduct experiments using ResNet-18 and ResNet-50 models, training on both the full datasets and our selected datasets. These models are then tested on more challenging benchmarks, including ImageNet-Hard Taesiri et al. (2024), ImageNet-R Hendrycks et al. (2021a), and ImageNet-A Hendrycks et al. (2021b). The results, shown in Table 5, demonstrate that models trained on our selected data consistently exhibit superior generalization and robustness on these harder ImageNet benchmarks compared to those trained on the original datasets. Notably, this improved performance is achieved with reduced training costs, further highlighting the efficacy of our approach.

Effect of Dataset Adaptation  To assess the effect of dataset adaptation, instead of using fine-tuned image and text adapters, we directly utilize the per-trained CLIP model to derive the scores $\bm{S}_{A}$ and $\bm{S}_{D}$. The experimental results, presented in Table 6 with a 90% selection ratio, show a significant decline in accuracy, with drops exceeding 2% on Tiny-ImageNet. Thus, dataset adaptation is essential for effectively transferring the model’s generalization ability to target datasets. This is particularly crucial for datasets that differ substantially from the pre-training datasets, such as CIFAR, where variations in image sizes and domain are distinct. Additionally, the adapters are implemented as MLP and fine-tuned with just 25 epochs, ensuring high efficiency.

Effect of Loss Terms  In Table 6, we evaluate the effect of each loss term and their combination in Eq. 6. The overall loss function achieves the highest accuracy. When $\mathcal{L}_{sa}$ is omitted, the selection process tends to prefer more diverse samples, but some class-representative samples may not be selected, which deteriorates the model performance severely. Without $\mathcal{L}_{sd}$, the selection emphasizes the most category-representative samples. Although the resulting performance drop is slightly smaller, the diversity of the selected datasets is compromised. Therefore, the incorporation of $\mathcal{L}_{sd}$ ensures a balanced representation of the selected dataset. Without $\mathcal{L}_{s}$, since we can not obtain the binarized selection decisions w.r.t. the expected selection ratios, we directly sort the scores in $\bm{d}$ and select the samples with higher scores. However, this degrades our method into a totally score-based selection and fails to address the group effect, leading to a noticeable drop in performance.