Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examples

The proposed framework ContRE provides a good practical estimator of DNN models’ generalization performance by measuring the loss/accuracy with contrastive examples as input. Specifically, given $m$ DNN models, we measure the Spearman’s rank correlation coefficient, noted as $r_{s}$ between two rank variables:

where $\text{cov}({\mathbf{X}},{\mathbf{Y}})$ is the covariance, $\sigma_{\mathbf{X}}$ and $\sigma_{\mathbf{Y}}$ are the standard deviations of the rank variables. Specifically, the rank variables are converted from $\{{\mathcal{L}}_{\bm{\theta}_{i}}({\bm{Z}}_{test})\}_{i=1,2,...,m}$ and $\{{\mathcal{L}}_{\bm{\theta}_{i}}({\bm{Z}}^{s})\}_{i=1,2,...,m}$.

We show in the experimental section that this strong correlation exists in various scenarios, across datasets, hyper-parameter settings, image transformation techniques, etc.

Only using correlation as the measure for association between two variables can be misleading because their dependence may come from the associations of each with a third confounding variable. The problem can be solved by controlling the confounding variable via partial correlation. Let ${\mathbf{X}}$, ${\mathbf{Y}}$ and ${\mathbf{Z}}$ be random variables taking real values, $r_{{\mathbf{X}}{\mathbf{Y}}}$ be Spearman correlation between ${\mathbf{X}}$ and ${\mathbf{Y}}$, then the partial rank correlation given control variable ${\mathbf{Z}}$ is:

We use the partial correlation to eliminate the effect of training accuracy $\{{\mathcal{L}}_{\bm{\theta}_{i}}({\bm{Z}})\}_{i=1,2,...,m}$ while investigating ContRE’s results. The details can be found in the experiment section.

To verify the existence of significant correlations on feature level, ContRE also proposes to investigate the clustering quality of intermediate features. The degree of separation of features from other classes can be measured elegantly by the Fisher ratio. Given $g$ different classes, there are $N_{i}$ data points in class $\pi_{i}$, with class mean $\bar{x_{i}}=\frac{1}{N_{i}}\sum_{j=1}^{Ni}x_{i,j}$. The between class scatter matrix measures mean separation and the within class matrix measures class concentration by pooling the estimates of the covariance matrices of each class, which are defined as:

Then the Fisher ratio is:

Fisher ratio is a powerful tool for analyzing models’ generalization capability right before the classifier. We compare features extracted from original samples and contrastive examples using Fisher ratio, i.e., $\{f^{l}({\bm{Z}};\bm{\theta}_{i})\}_{i=1,2,...,m}$ and $\{f^{l}({\bm{Z}}^{s};\bm{\theta}_{i})\}_{i=1,2,...,m}$, as part of explanations of why ContRE works well. The detailed findings are located in the experiment section.

Three of the most popular benchmark datasets are considered: CIFAR-10, CIFAR-100 and ImageNet. CIFAR datasets contain 60,000 tiny images of resolution 32$\times$32 while ImageNet includes over 1.2M natural images. Conventional split of training and test sets are followed for CIFAR datasets, while the validation set of ImageNet is used as testing set here.

For evaluating our proposed framework ContRE, the most popular or the most powerful DNN architectures are considered: VGGNet (Simonyan and Zisserman, 2015), ResNet (He et al., 2016), Wide ResNet (Zagoruyko and Komodakis, 2016), DenseNet (Huang et al., 2017), ResNeXt (Xie et al., 2017), ShuffleNet (Zhang et al., 2018; Ma et al., 2018), MNasNet (Tan et al., 2019), MobileNet (Howard et al., 2017; Sandler et al., 2018) and their variants. We train these DNN models with standard data augmentation methods, i.e., random crop and random flip, and standard training hyper-parameters. We also investigate the different data augmentation methods and different hyper-parameter settings for training DNN models, and the strong correlation between ${\mathcal{L}}_{\bm{\theta}}(R({\bm{Z}}))$ and ${\mathcal{L}}_{\bm{\theta}}({\bm{Z}}_{test})$ still holds. Generally, models for CIFAR datasets are smaller than those for ImageNet but the architectures are shared among models for these datasets. Moreover, many pretrained models on ImageNet are publicly available²²2https://pytorch.org/vision/stable/models.html, which greatly accelerate our setup process.

As introduced previously, RandAugment (Cubuk et al., 2020) is used in the ContRE framework to generate contrastive examples, where we take all the available transformation operations from the open-sourced implementation³³3https://github.com/ildoonet/pytorch-randaugment. We note RA_N$X$_M$Y$, where $X$ and $Y$ are the two hyper-parameters to configure in RandAugment: the number of sequential distortion operations and the magnitude of each distortion. For example, RA_N2_M9 means that 2 operations will be chosen randomly from all the available operations and the distortion of magnitude 9 will be applied⁴⁴4The maximum magnitude is 30. Note that even the maximum magnitude does not transform the image into a total noise..

In our framework ContRE, two hyper-parameters in RandAugment can be tuned: $N$ denotes the number of transformations that are to be applied on the original examples, and $M$ denotes the magnitude of distortions. Though they both represent the amount of distortions, their effects on ContRE are varied. As shown in 2, we have tested all the compositions of $N=\{2,3,4\}$ and $M=\{4,9,15,20\}$. A large ($M=15$) magnitude of each distortion leads to a higher correlation but an additional increase from 15 to 20 does not change much. The results match our previous observations, implying that a weak distortion on the samples cannot help to tell good classifiers from the bad ones. On the other hand, applying more transformations at the same time seems to hurt the performance, especially when $M$ is small. Therefore, our approach adapts a large value of $M=20$ and a smaller value of $N=2$, which is proved to be quite robust across various tasks and datasets.

ContRE generalizes the idea of contrastive learning to the evaluations of DNN models’ robustness and generalization performance. However, is it really necessary to use a stochastic module for generating the contrastive examples for ContRE? Is there any single transformation that could achieve the desired results? We answer these questions through the comparison of the effects by ContRE and each single contrastive technique.

Following the same experimental setup, contrastive examples are generated under each single transformation with the same magnitude as ContRE and the experiments are performed using all DNN models considered previously. The obtained Spearman correlation coefficients range from -0.297 to 0.988. While a part of single transformations yield correlations higher than 0.9, especially for ImageNet-trained models, there is no single operation that performs well across the three datasets. Therefore, choosing a best transformation depends on the dataset and it is difficult to find a single transformation that works well across different datasets. In contrast, ContRE introduces robustness and consistently achieves high performance through the expectation over the stochastic module and by combining several competent candidates.

To further prove the effectiveness and efficiency of ContRE, we conduct similar experiments on CIFAR datasets using contrastive examples that are generated by all the possible compositions of two transformations, obtain the accuracy on various contrastive examples and show the Spearman correlations with the testing accuracy in Figure 4. Some observations are shared with the results from the comparison using single transformations, that any composition with “Posterize” does not work well for CIFAR datasets and that any composition with “Rotate” works well for CIFAR-10 but not for CIFAR-100. A small part of compositions obtain higher correlations than ContRE. We argue that searching the optimal composition demands much computation resource and the optimal choice varies from different datasets, while ContRE benefits the expectation over the stochastic module to consistently produce precise estimation results. We will present in the following that ContRE is consistently applicable, efficient and effective in many practical complex situations.

A direct application of ContRE is to find the best model among models with the same architecture but trained with different hyper-parameters. We have trained 18 models, all of them is of the network structure of Wide-ResNet-26-10, on CIFAR-10 with various choices of batch size (32, 64, 128), learning rate (0.01, 0.05, 0.1) and weight decay (0.0001, 0.001). Based on the resulting 18 models, ContRE outperforms other approaches by achieving a correlation coefficient Corr of 0.859, see the results in Figure 5 (left) and the comparison with other approaches in Table 3. The results illustrate that, besides helping with selecting the best architecture, our approach can be helpful in the hyper-parameter-tuning stage.

In some practical settings, training data are very scarce while the experiments we have shown are using a large dataset with over 100 samples per class. So we here would like to test the robustness of ContRE by reducing the training data size to small fraction (20%) and training the DNN models. When provided with 20% training data, models have limited information of the true data distributions and the generalization performances drop by 3-10 percent. Under such setting, ContRE outperforms trivial approaches by achieving a correlation coefficient of 0.754. The results indicate the potential of ContRE in predicting model performances when models are not trained with abundant data.

Given ContRE’s high correlations between the generalization performance and the accuracy on contrastive examples of DNN models that are trained with standard data augmentation methods, we have observed that the “Random Crop” and “Random Flip” that are used for the standard training process do not provide desired results in our framework. For over-parametric DNN models, including more data augmentation methods could usually improve the generalization performance. We thus ask if the correlation obtained by ContRE mainly benefits from the image transformation methods that are not used during training and whether ContRE still performs well when the generated contrastive examples have already been used during training.

To answer the questions, we have trained the same set of DNN models on CIFAR-10 with replacing the standard data augmentation by the RandAugment module while leaving other training settings unchanged. Then, we repeat the processing of ContRE and get that ContRE still achieves a correlation higher than 0.9, with the results in Figure 5 (right) and comparisons in Table 3. We note that “Random Flip” and “Random Rotation” get higher correlations than ContRE. We argue that single transformations could get high correlations if they are not used during training, but the correlation coefficients largely decrease if they have been used during training, as shown in the previous results. In contrast, ContRE combines a number of single transformations to generate contrastive examples and introduces the robustness from the expectation over the stochastic module. Even with the models that have seen the contrastive examples during the training stage, ContRE can also successfully estimate the generalization performances.

ContRE directly estimates the generalization performance instead of the generalization gap (the difference between the training accuracy and the generalization performance). In the previous experiments, we mainly discussed the estimations of generalization performance in this work. Meanwhile, under the general condition that all DNN models equally fit on the training set (Jiang et al., 2020b), ContRE is also capable of providing a good estimator of the gap.

To further demonstrate the feasibility of ContRE, we use the experiment settings of “Predicting Generalization in Deep Learning Competition” at NeurIPS 2020 (Jiang et al., 2020a) to evaluate ContRE on predicting the generalization gap of DNN models using the contrastive examples of the training set. The competition offers a large number of deep models trained with various hyper-parameters and DNN architectures trained on CIFAR-10 or SVHN, while the evaluator of the competition first estimates the generalization performance of every model using the proposed measure, then computes the mutual information score (the higher the better) between the proposed measures and the (observable) ground truth of generalization gaps.

In the experiments, we propose to use the difference between the accuracy based on the original training samples and the one using contrastive examples generated from the original training samples with ContRE. For the comparison reason, we include several baseline measures in generalization gap predictors, including VC Dimension (Vapnik, 2013), Jacobian norm w.r.t intermediate layers (Jiang et al., [n.d.]), distance from the convergence point to initialization (Nagarajan and Kolter, 2019), and the sharpness of convergence point (Jiang et al., 2019b), all of these baselines provide a theoretical bound for the estimated gap. Table 4 presents the comparisons between the proposed measures and baselines. It shows that the proposed ContRE framework, with estimating the gap between the accuracy on original training samples and the one on contrastive examples, significantly outperforms the baseline methods. Note that our proposed approach ContRE is an efficient and effective estimator for the generalization performance in practice, and that ContRE is a good estimator for the generalization gap in condition that the training accuracy be controlled on the same level.

We observe that the training accuracy is well correlated to the testing accuracy on ImageNet, where the best model cannot fit the training set. This shows that the training accuracy might be a good estimator of the generalization performance in some cases. On the other hand, training accuracy is also related to the accuracy on contrastive examples because they are directly generated from the training set. A question naturally follows: is the training accuracy the key factor for ContRE being a good estimator?

To answer this question, we first define the notion of consistency ${\bm{C}}_{\bm{\theta}}({\bm{Z}}^{s})$ as the difference between training accuracy and contrastive examples based accuracy on the training set ${\bm{C}}_{\bm{\theta}}({\bm{Z}}^{s})={\mathcal{L}}_{\bm{\theta}}({\bm{Z}})-{\mathcal{L}}_{\bm{\theta}}({\bm{Z}}^{s})$. The testing accuracy and the accuracy on contrastive examples contain a common variable, i.e., training accuracy ${\mathcal{L}}_{\bm{\theta}}({\bm{Z}})$, shown in the decomposition below:

With controlling the effect of this compound variable, training accuracy ${\mathcal{L}}_{\bm{\theta}}({\bm{Z}})$, we compute the partial correlation between the accuracy on contrastive examples ${\mathcal{L}}_{\bm{\theta}}({\bm{Z}}^{s})$ and the testing accuracy ${\mathcal{L}}_{\bm{\theta}}({\bm{Z}}_{test})$. Directly estimating the gap with the consistency fails on ImageNet for the reason that models fit to ImageNet training set in varying degrees. This is not a practical issue for two reasons: (1) it will be less needed to estimate the gap if the generalization performance is strongly correlated to the training accuracy; (2) few datasets are in the scale of ImageNet. Nevertheless, the results in Table 5 show ContRE still gets relatively strong (partial) correlations on all the three datasets, while a moderate difference is observed from the comparison between CIFAR datasets and ImageNet. We note that in the case of ImageNet, where the best model cannot perfectly fit the training set, the training accuracy is directly correlated to the testing accuracy and plays an important role for generalization performance estimation. However, for CIFAR datasets, where all (or some) models get 100% training accuracy, the training accuracy loses the predictability and the partial correlations controlling the training accuracy have no large difference compared to the standard correlations. In summary, the training accuracy is not the key factor that ContRE works well for the generalization performance estimation in most cases.

For complex DNN models, we further conduct analysis experiments by looking into the intermediate features of DNN models, computed on the original and contrastive examples. For these examples, we compute the Fisher ratios of features $f({\bm{Z}};\bm{\theta})$ before the classifier for all models, and get the Spearman correlations with test accuracies. Note that for ImageNet and CIFAR-100 datasets, the number of data instances in each class is less than the number of dimensions of feature vectors when the DNN model, for instance DenseNet, is wide, leading to the non-invertibility of the within class matrix during the computation of Fisher ratio. We therefore perform the dimension reduction techniques on the feature vectors using singular value decomposition (SVD), reducing the number of dimensions to 64 or 512 for CIFAR-100 or ImageNet respectively, with most models containing over 95% of the total variance.

The results in Figure 6 (a) show that Fisher ratios from ContRE are highly correlated with the generalization performance of DNN models on each of the three datasets. We also conduct experiments to compute the results on models with the same architecture but different hyper-parameters, shown in Figure 6 (b). The conclusion also holds for models trained with different hyper-parameters. The overall numerical comparison is shown in Table 6. It suggests that, among contrastive examples based on various operations, models that are able to generate better class-separated features from ContRE samples are more likely to have better generalization performance. Intuitively, Fisher ratio, which can be seen as a measure of the clustering quality, is highly correlated with the final classification accuracies. Therefore, the high correlation between ContRE accuracies and generalization performance inherently comes from the association between ContRE accuracies and its features’ degree of separation, and that between Fisher ratio of features and model generalization performance.