Exploration of Class Center for Fine-Grained Visual Classification

Benefiting from the development of deep learning [31, 32], there has been significant progress in existing FGVC research in recent years [33, 34, 35]. FGVC methods can be divided into two categories based on whether they employ extra manual annotations: strongly supervised methods and weakly supervised methods. Strongly supervised methods require manually labelled bounding boxes or part annotations, with which informative key parts can be located for extracting discriminative part features [36, 37, 38, 39, 40]. Finally, key part features and object features are integrated for classification. However, these additional manual annotations require extensive expert knowledge. There is limited feasibility and scalability in real-world applications. Therefore, weakly supervised methods without manual annotation have attracted much attention from researchers [34, 35, 41, 42, 43]. M2DRL[44] learns multigranular discriminative region attention and multiscale region-based feature representation for more accurate object region positioning and category recognition. DME-Net[34] introduces a multitasking framework for the low-resolution fine-grained image recognition task, that aims to capture reliable object descriptions from macro- and microperspectives, respectively. SIM-Trans[45] incorporates object structure information into transformer to enhance discriminative representation learning to contain both appearance information and structure information. SIA-Net[35] extracts the low-level image details under the guidance of accurate semantics and makes the details spatially correspond to high-level semantics with complementary content. AA-Trans[11] acquires discriminative parts of the image precisely to better capture local fine-grained information. MA-CNN[6] locates part localization with proposed channel grouping layers in a weakly supervised manner. Then, part-based features and object-based representations are integrated to produce the final classification. MAMC [10] and Cross-X [9] obtain specific part features directly through attention mechanisms with an end-to-end network. In addition, PMG [46], CA-PMG [47] and DCL [48] force models learn specific features from jigsaw patches. Zhang et al.[41] leverages a small and clean meta-set to provide reliable prior knowledge for tackling noisy web images for webly-supervised FGVC. MetaIRNet[42] combines generated images with original images to generate hybrid training images to improve the performance of one-shot FGVC. The above weakly supervised methods employ complicated structures and complex training strategies to extract key part features to reduce intra-class variances. However, there is no efficient way to handle inter-class differences in FGVC.

The concept of class center is introduced to represent the whole class by center loss[12, 49], in which Euclidean distances between sample features and class centers are minimized to enhance the discriminative features in neural networks. Furthermore, Farzaneh et al.[14] argued that not all elements in a class-center feature are relevant to discrimination, and further proposes sparse center loss. Specifically, the sparse center loss is calculated by multiplying the Euclidean distance in the center loss by the weights from an attention network. Li et al. also referred to the idea of class center and proposed single center loss (SCL)[15]. SCL aggregates representations of natural samples around the center point and increases the distance from manipulated samples to the center point, making it greater than from natural samples by a margin. CSDL[13] combines class centers and one-hot labels to generate soft labels. To measure the importance of samples in the same cluster, AdaMG[50] calculates the distances between these samples and their corresponding class centers. Some few-shot methods represent the class as a whole with prototype [51, 52, 53], which is similar to the class center. However, considering intra-class variances, these methods with class center do not address the inter-class differences of FGVC, and do not fully explore for the ability of class centers.

One-hot labels are eligible for coarse-grained visual classification because of significant visual differences between coarse categories[13]. However for FGVC, models with hard labels pay attention to irrelevant features (e.g., background) or sample-specific noise to achieve high prediction confidence from these ”hard” labels. LS[23] reduces prediction confidence by uniformly redistributing partial probabilities to nontarget classes to produce smoothed soft labels. Local distributional smoothness (LDS)[54] proposes the local distributional smoothness of model outputs as a regularization term when inputs are perturbed. LR[25] and GLR[26] consider the rich inter-class correlations of FGVC, and optimize models with instance-level soft labels generated from a trained teacher network. However, the soft labels in these methods may be incorrect.

MCC optimizes feature distances between sample features and multiple class-center features, and aims to handle intra-class variances and inter-class differences. Given that the $i$-$th$ image belongs to the $y_{i}$-$th$ class, a basic neural network is utilized as a backbone to extract the $D$-dim feature $X_{i}\in\mathbb{R}^{D}$.

First, class-center features $F_{y_{i}}\in\mathbb{R}^{D}$ are initialized and updated as the representation of the $y_{i}$-$th$ whole class. In most existing loss functions based on class-center [12, 14], class-center features are set as learnable parameters, and are updated backpropagation. Because of the small number of samples in each class of FGVC, it is difficult to stably learn robust class-center features. To address this issue, the MCC updates the class-center feature $F_{y_{i}}$ by averaging all the input sample features of the $y_{i}$-$th$ class in the training phase. Specifically, we first restore the sum of previously inputted sample features from the current class-center feature $F_{y_{i}}^{cur}$ of the $y_{i}$-$th$ class. For this purpose, a counter $C_{y_{i}}$ is maintained to record the number of sample features used for updating $F_{y_{i}}$. The sum of the previously input sample features of the $y_{i}$-$th$ class can be obtained by multiplying the current counter $C_{y_{i}}^{cur}$ by $F_{y_{i}}^{cur}$. Then, a new sample feature $X_{i}$ is added to the sum to update $F_{y_{i}}^{cur}$ as $F_{y_{i}}$. The updating of $F_{y_{i}}^{cur}$ and $C_{y_{i}}^{cur}$ is formulated as follows:

As the average of features of all samples in the $y_{i}$-$th$ class, $F_{y_{i}}$ does not need to be updated as learnable parameters in the training phase. Thus the updating is quite stable. The stability and effectiveness of our strategy are demonstrated in ablation studies.

Then we constrain the intra-class variances and inter-class differences with class-center features. To constrain intra-class variances, we reduce the feature distance between sample feature $X_{i}$ and the corresponding class-center feature $F_{y_{i}}$ directly. However, constraining only intra-class variances is not enough for FGVC. Subtle inter-class differences still cause confusion. Thus, for inter-class differences, we enlarge feature distance between $X_{i}$ and $F_{sim_{y_{i}}}$, which is the most similar class-center feature to $F_{y_{i}}$. $F_{sim_{y_{i}}}$ is obtained according to the similarity of class-center features. Specifically, we first construct a cosine similarity matrix $S\in\mathbb{R}^{N\times N}$, in which the similarity $s_{h,w}$ between the $h$-$th$ class and the $w$-$th$ class is calculated as

$N$ denotes the number of classes. With the cosine similarity matrix $S$, the most similar class to the $y_{i}$-$th$ class is chosen by searching for the maximum value of the $y_{i}$-$th$ row in $S$:

where $sim_{y_{i}}$ is considered the index of the most similar class to the ${y_{i}}$-$th$ class.

In practice, the cosine distance $D_{cos}$ is utilized to represent the feature distance. Normally, $D_{cos}(X_{i},F_{y_{i}})$ should be minimized in the trained phase, while $D_{cos}\left(X_{i},F_{sim_{y_{i}}}\right)$ should be maximized. In order to integrate these two distances into a loss, cosine distance $D_{cos}\left(X_{i},F_{sim_{y_{i}}}\right)$ is replaced with the cosine similarity $cos\left(X_{i},F_{sim_{y_{i}}}\right)$. Moreover, $cos\left(X_{i},F_{sim_{y_{i}}}\right)$ is multiplied by similarity $s_{y_{i},sim_{y_{i}}}$ as a weight to adaptively handle confusing classes. For a mini-batch, the MCC loss function is expressed as

where $M$ denotes the number of samples in a mini-batch, $cos$ is the cosine similarity, and $k$ is the index of the sample in the mini-batch.

In this section, CLG is proposed to generate proper soft labels to alleviate overfitting and introduce relationships among classes. In our method, soft labels are generated from class-center distributions. Like class-center features, the class-center distribution $L_{y_{i}}\in\mathbb{R}^{N\times 1}$ of the $y_{i}$-$th$ class is generated by averaging distributions of all input samples in the $y_{i}$-$th$ class during training phase. The updating strategy of the CLG can be expressed as follows:

where $f$ represents the last fully connected layer, which outputs a $N$-dim vector. Then, the sample probability distribution $P_{X_{i}}$ and the class-center probability distribution $Q_{y_{i}}$ are calculated with the $softmax$ activation function.

Consequently, the class-center probability distribution $Q_{y_{i}}$ is regarded as the soft label of $X_{i}$. The nontarget categories that are more similar to the target category have higher confidence in the soft labels. This nonuniform confidence is more reasonable and realistic.

For a mini-batch, KL divergence is computed between sample probability distributions and soft labels as the CLG loss:

$p_{k,n}$ and $q_{y_{k},n}$ denote the $n$-$th$ elements in $P_{X_{k}}$ and $Q_{y_{k}}$, respectively. With models supervised by soft labels of CLG, overfitting is effectively alleviated. Moreover, rich information among categories is considered to further improve the ability to model objects.

Finally, the MCC and CLG are integrated as the ECC, and the two components are multiplied by the hyperparameters $\lambda_{1}$ and $\lambda_{2}$, respectively, to adjust the effects for model training. The entire ECC is formulated as,

In addition, the CE loss $L_{CE}$ is combined with our ECC. The final loss function is expressed as follows:

where $l_{k,n}$ denotes the $n$-$th$ element in the one-hot label of $X_{k}$, and $d_{k,n}$ is the $n$-$th$ element in the sample distribution $f(X_{k})$.

Four widely-used fine-grained datasets are utilized in our experiments, including the AIR[27], CUB[28], CAR[29] and NAB[30] datasets. The details of datasets are shown in Table.I. In addition, we conduct the experiments on a large-scale dataset iNaturalist 2018 (iNat2018)[55]. The experimental results of iNat2018 are displayed and discussed in the supplementary material.

ResNet50[31] is used as backbone in our experiments unless otherwise stated. For data augmentation, the images are resized to 600×600. Random cropping and center cropping are utilized to crop image to 448×448 during the training and test phases respectively. In addition, we apply random horizontal flipping in training. Stochastic Gradient Descent (SGD) is utilized with a momentum of 0.9. The initial learning rate is 0.01, which decays every 15 epochs at a decay rate of 0.1. The initial learning rate is multiplied by 0.1 for the pretrained backbone on the CUB dataset. The batch size and epochs are set as 32 and 50, respectively. Class-center features and class-center distributions are randomly initialized before training. For the center loss, PC loss [56] and LS, we choose the optimal hyperparameters among the settings from original papers and our experimental best values. Any extra annotations or extra training data are not used and all backbone models are pretrained on the ImageNet dataset.

In ablation studies, same training hyperparameters (including batch size, learning rate and so on) are utilized. And Resnet50 is used as the backbone for all ablation experiments.

First, to verify the effectiveness of our method, we test it on different backbones including InceptionV3[23], ResNet50, ResNet101[31] and DenseNet121 [57]. According to Table.II, our ECC brings satisfactory improvements on four datasets (0.8%$\sim$1.9% on AIR, 0.6%$\sim$3.4% on CUB, 1.0%$\sim$1.6% on CAR and 1.2%$\sim$3% on NAB). Compared with the improvements on InceptionV3 (0.8% on AIR, 0.6% on CUB, 1.2% on CAR, 1.2% on NAB), ResNet50, DenseNet121 and ResNet101 have better feature extraction capabilities for constructing better class centers. Thus, there are more improvements on those models (average boost of 1.83% on AIR, 2.8% on CUB, 1.4% on CAR and 2.9% on NAB).

We also show the results of integration with existing FGVC methods, including DCL[48], MGE-CNN[4], WSDAN[2], Swin transformer [58] and CAL[59] in Table.III. Compared with baselines, integrated methods obtain obvious and consistent improvements on these four datasets.

In this section, we compare our approach with different loss functions, including Center loss (Ct loss)[12], Single Center loss (SC loss)[15], Pairwise Confusion loss (PC loss)[56], Label Smoothing (LS)[23], Contrastive loss (Ctt loss)[18] and Triplet loss (Tlt loss)[19] on four datasets. For fairness, experiments are conducted with different hyperparameters (including recommended hyperparameters from original papers and our experimental best values) on ResNet50 to choose the optimal results for comparison with our methods. Moreover, ResNet50 is replaced with different backbones (Inceptionv3, DenseNet121, ResNet101) to further demonstrate the superiority of our method. Results are shown in Table.IV. CE loss is regarded as baseline, which have achieved acceptable results. Existing methods based on class center (Ct loss, SC loss) bring few improvement, due to the unstable updating strategy of class centers and the lack of constraints for inter-class differences. Contrastive loss and Triplet loss (Ctt loss and Tlt loss) achieve better performances than methods based on class center. Compared with the above methods, our method effectively constrains intra-class variances and inter-class differences and achieves optimal overall performance.

An exception is InceptionV3 on CUB dataset, where Ctt loss and Tlt loss are superior to our MCC (85.1% vs 84.5%). In fact, in our method, the quality of the class-center features and class-center distributions depends on the quality of the sample features and sample distributions extracted from the backbones. Compared with other backbones (ResNet50, ResNet101 and DenseNet121), the features and distributions from InceptionV3 are not good enough, which leads to limited improvement. In practice, compared with InceptionV3, ResNet and DenseNet have more extensive applications, and most FGVC methods use ResNet and DenseNet as backbones. With these backbones, our method has more obvious advantages. Therefore, our MCC loss has greater application value in FGVC.

In this section, integrated models are compared with existing state-of-the-art (SoTA) approaches. Extensive experiments are conducted to verify the effectiveness of our ECC. Table.V shows the performances. Compared with other methods, our ECC (Swin-ECC on the CUB dataset, CAL-ECC on the AIR and CAR datasets, and Swin-ECC on the NAB dataset) achieves excellent performances, with SoTA on the CUB and AIR datasets. Although ALIGN outperforms our method on CAR dataset, ALIGN is pretrained on a large-scale noisy image-text dataset (LSNITD)[63], which includes 1.8B image-text pairs. ImageNet-1k and ImageNet-21k are utilized in our integrated models. The size of LSNITD is 120 times larger than ImageNet-21k.

Moreover, we made an interesting observation. Transformer-based methods achieve better results than CNNs on the CUB and NAB datasets, but CNNs achieve better performances on the AIR and CAR datasets. We argue that transformer-based methods are naturally not subject to the local inductive bias of CNNs. Thus, these methods have the ability to model global dependency, which has advantages over CNN-based methods in terms of classifying nonstructural rigid objects (e.g., birds). However, transformer-based methods destroy the structural information of rigid structural objects including cars and aircraft, by preprocessing an image into a sequence of flattened patches [65]. This process leads to inferior performance on the AIR and CAR datasets.

In this section, ablation studies are conducted for different components of our ECC. First, hyperparameters of the MCC and CLG are investigated via extensive experiments. Then, each component in our approach is explored. In addition, we discuss different updating strategies of class centers in detail.

In this section, we consider the complexity of our ECC. First, the time complexity is discussed. To update the class center feature, the MCC needs to index the class center feature with the corresponding label. The class-center feature is updated according to Eq. (1). The time complexity of updating the class center feature is $O(1)$. To handle inter-class differences, the MCC needs to calculate the similarity between each class according to Eq. (3), whose time complexity is $O(N^{2})$, where $N$ is the number of classes. Then, the MCC algorithm searches the maximum in a row of the similarity matrix, with a time complexity of $O(N)$. Finally, the MCC loss is calculated according to Eq. (5) with a time complexity of $O(1)$. In summary, the time complexity of MCC is $O(1)+O(N^{2})+O(N)+O(1)\approx O(N^{2})$. Furthermore, the CLG indexes the class center label with the corresponding label to update the class center label according to Eq. (6) and calculates the loss according to Eq. (9). The time complexity of the CLG is $O(1)$. The overall time complexity is determined by the complexity of the MCC and the CLG. Therefore, the overall time complexity of our method is $O(N^{2})+O(1)\approx O(N^{2})$.

Second, we discuss the computational complexity and calculate the FLOPs of the MCC and CLG. Actually, the computational complexity of the CLG is less than that of 0.1 G FLOPs, which is negligible considering that of the FLOPs of the MCC. Therefore, we regard the FLOPs of the MCC as the overall FLOPs. The comparison of the computational complexity before and after using our method is shown in Table.VIII. Our method requires only a few FLOPs (average 0.06 G to 1.66 G on all datasets). Compared with the complexity of the backbone network, this computational complexity is insignificant. Moreover, the costs exist in the training phase only and are not incurred in the practical test phase. Thus, our method is very practical and can result in significant improvement with negligible costs.

First, we visualize heatmaps with Grad-CAM[68] images, which are shown in Fig.8. Our ECC guides the model to learn discriminative features and alleviates model overfitting. It is evident that the model no longer pays attention to background information, especially in the CUB and NAB datasets, which usually contain complex backgrounds. Moreover, in the first and last heatmaps of the AIR dataset, the results of CE loss incorrectly focus on complex background information, while our method correctly focuses on the objects. Similarly, our method achieves better results on the CAR dataset. Fig.9 displays the t-SNE results of the baseline and our ECC loss on 18 species of visually similar warblers from the CUB dataset. The left image and right image represent the results of CE loss and our ECC loss, respectively. There are more evident margins between different classes in the results of ECC than in those of CE loss. Compression within a class is also observed in the t-SNE results. These results indicate the effectiveness of our method.