Webly Supervised Fine-Grained Recognition:
Benchmark Datasets and An Approach

Recent success of deep learning has shown that a deep network in conjunction with abundant well-labeled training data is the most promising approach for fine-grained recognition [1, 2, 19, 20]. However, even with the availability of scalable crowd-sourcing platforms like Amazon Mechanical Turk, constructing a large-scale fine-grained dataset like iNat2017 [4] is still an extremely difficult work since distinguishing subtle differences among fine-grained categories (e.g., different animals [5, 11], or plants [12, 17]) usually requires domain-specific expert knowledge.

To reduce the cost of manual fine-grained annotation, many methods have been proposed, which are primarily focused on semi-supervised learning [18]. These works inevitably involve various forms of human intervention and remain labor-consuming [26]. To further reduce manual annotations as well as to learn more practical fine-grained models, training directly from web images is becoming increasingly popular [13, 14, 15, 16]. Nevertheless, the lacking of a benchmark dataset makes it difficult to fairly compare the performances of these algorithms. This is the motivation of this work and we aim to provide a benchmark dataset for evaluating webly supervised fine-grained recognition algorithms.

Webly supervised fine-grained recognition consists of three challenges: (1) Label noise - Different from manually labeled datasets, fine-grained web images are often associated with label noises due to the influence of error-prone automatic or non-domain-expert annotations. We consider two types of label noises, which we call “cross-domain” and “cross-category” noises. To be specific, cross-domain noise is the portion of images that are not of any categories in the same fine-grained domain, e.g., for birds, it is the fraction of images that do not contain a bird (cf. images with purple bounding boxes in Fig. 1 (a)). In contrast, cross-category noise is the portion of images that have the wrong labels within a fine-grained class, e.g., an image of a bird with the wrong category label (cf. images with red bounding boxes in Fig. 1 (a)). According to existing works [24], deep neural networks have memorization effects, which will memorize incorrectly labeled training data and cause a poor generalization performance [25]. (2) Small inter-class variance - As shown in Fig. 1 (b), three fine-grained sub-categories have small inter-class variance while each sub-category has large intra-class variance. Aiming at recognizing hundreds of sub-categories belonging to the same super-category is an extremely challenging task. (3) Class imbalance - The natural world is heavily imbalanced, as some species are much more likely to be observed [4]. As shown in Fig. 1 (c), we collected 563 web training images for “Udea Rubigalis”. While for “Hordnia Atropunctata”, we only gathered 4 web images for training. Extreme class imbalance is a property of the real world fine-grained categories and recognition models should can deal with it.

To address aforementioned issues, we leverage the categories in three famous fine-grained datasets, i.e., FGVC-Aircraft [27], CUB200-2011 [3] and Stanford Cars [28], to construct a new webly supervised fine-grained benchmark dataset WebFG-496. To scale up the categories as well as to build a much larger and challenging fine-grained dataset, we reuse the categories in iNat2017 [4] to build a web version iNaturalist dataset, termed as WebiNat-5089. Compared to manually labeled iNat2017, WebiNat-5089 consists of 1,184,520 web training images, basically twice the number of training images (i.e., 579,184) in the original iNat2017.

Furthermore, we also propose a simple yet effective learning paradigm to train robust deep fine-grained models directly from noisy web images. Our work is motivated by the following observations: 1) The performance of deep neural networks can be boosted by learning from wrongly classified instances (i.e., “hard examples”) [29]. 2) Deep neural networks always fit to “easy examples” first, and gradually adapt to “hard examples” [24]. 3) Distinct networks have different learning abilities, and can collaboratively boost their performance by mutually communicating “useful information” [30]. Specifically, we train two deep neural networks simultaneously and let them mutually correct their classification errors. For each mini-batch of web images, each network individually feeds forward all data to separately predict the labels, based on which the input data is split into two groups $G_{d}$ (instances with different predictions) and $G_{s}$ (instances with identical predictions). Then, the networks update their parameters with the instances in $G_{d}$. Meanwhile, each network sorts and fetches small-loss instances in $G_{s}$ as “useful knowledge”, and communicates this “useful knowledge” with its peer network for correcting the errors induced by instances in $G_{d}$ during updating. Extensive experiments on the new benchmark datasets demonstrate the effectiveness of our proposed approach. The main contributions of this work can be summarized as follows:

In this section, we explain the construction details of our webly supervised fine-grained datasets WebFG-496 and WebiNat-5089. It should be noted that WebFG-496 consists of three sub-datasets: Web-aircraft, Web-bird, and Web-car.

Fine-Grained Categories: The first issue for building a new benchmark dataset is the fine-grained categories of web images we shall collect from the Internet. In the literature, there are three famous manually labeled fine-grained datasets, i.e., FGVC-Aircraft [27], CUB200-2011 [3], and Stanford Cars [28], which contain 100 types of airplanes, 200 species of birds, and 196 categories of cars, respectively. For our WebFG-496, we follow these fine-grained datasets to reuse the category labels of them as our target fine-grained categories. Furthermore, to construct a much larger and more challenging webly supervised fine-grained benchmark dataset, we explore the 5089 categories originally presented in iNat2017 [4] to build our WebiNat-5089.

Testing Images: Since the fine-grained categories in our webly supervised datasets WebFG-496 and WebiNat-5089 come from the existing datasets as aforementioned. To save the cost of dataset construction and consider convenient comparisons with traditional fine-grained methods, we directly take the testing sets in FGVC-Aircraft, CUB200-2011, and Stanford Cars as the testing data for our WebFG-496. For WebiNat-5089, the validation set of iNat2017 is utilized as the testing data.

Web Sources: As pointed out by [51], different web sources like Google Image Search Engine (GIS), Bing Image Search Engine (BIS), Flickr, Airliners, and iNaturalist may have a significant influence on the datasets. Table 2 summarizes the web sources for testing images in our WebFG-496 and WebiNat-5089. To reduce the probability of overlap with the testing set along with to train webly supervised domain robust deep fine-grained models, we ultimately choose Bing Image Search Engine (BIS) as our web source of training images.

Collecting Candidate Training Images: Top-ranking images of image search engines tend to have relatively high accuracy and lower-ranking images usually contain more and more noises. Based on the consideration of reducing noise in the collected web images, for the 496 categories in WebFG-496, we treat each category label as a query and crawl the top 150 images from the BIS. Since some special categories are much harder to photograph than others, the natural world is extremely imbalanced [4] and the realistic data obeys long-tailed distribution. For the 5089 categories in WebiNat-5089, we also treat each category label as a query but crawl as many images as possible for each query. After removing the invalid links, we obtained 71,987 images for WebFG-496 (14,817 aircraft images, 29,211 bird images, and 27,959 car images) and 1,437,483 images for WebiNat-5089, respectively.

Removing Broken Images: Since our training images are directly crawled from the web, some broken images may be included. To remove these broken images, we employ the Python library Pillow to check each collected image and subsequently convert them to the RGB mode. Images that cannot be opened by Pillow or cannot be converted to RGB will be regarded as broken ones and get deleted. Table 2 gives a detailed number of images for WebFG-496 and WebiNat-5089 after broken images removing.

Removing Duplicated Images: To remove duplicated images between training data and testing data, we take advantage of deep Convolutional Neural Network (CNN) in semantic information extraction. Our overlap removing strategy is under the assumption that images with more similar semantic information are more likely to be similar or even identical. Specifically, we first use the VGG-16 [57] model pre-trained on ImageNet to extract the embedding feature vector for each image in both training and testing data. Here we select the feature maps of the last max-pooling layer and then perform global average pooling to cast them into a 512-d feature vector. Then, for every single test image per category, we calculate the similarity distance between this testing image and every training image. For each category, we obtain the smallest distance between training and testing data, which is denoted as $\theta$. We set an empirical threshold factor $\eta$ to scale the distance to $(1+\eta)\times\theta$ and remove the web training images which have a smaller distance than $(1+\eta)\times\theta$. In dataset constructions, the value of $\eta$ is set to be 0.01. As shown in Table 2, after duplicated images removing, we ultimately obtain 53,339 training images for WebFG-496, and 1,184,520 training images for WebiNat-5089. After that, we manually check the obtained images according to the ranked distance between training and test data, and few duplicated images are left.

Class Imbalance in WebiNat-5089: WebiNat-5089 contains 2,101 types of plants, 1,021 kinds of insects, 289 species of reptiles, etc. The average number of images per class for the WebiNat-5089 dataset is 232.7 while the median number is 221. The images for some categories are easy to obtain (e.g., 563 images for “Udea Rubigalis”) while others are hard (e.g., 4 images for “Hordnia Atropunctata”), making the extreme class imbalance a property (i.e., long-tailed) for our WebiNat-5089 dataset. Therefore, training fine-grained models from web images also needs to take the class imbalance problem into account.

Dataset Accuracy: It is difficult to manually establish the accuracy of fine-grained web data, especially for WebiNat-5089 which contains over 1.1 million web training images. However, we can roughly estimate the accuracy of training data for WebFG-496 and WebiNat-5089 by random sampling. For WebFG-496, we randomly select 100 sub-categories and 50 images for each sub-category. For WebiNat-5089, we randomly select 200 sub-categories and 50 images for each sub-category. Finally, the roughly estimated accuracy of training data is shown in Table 3.

Training strategy As shown in Fig. 2, our framework consists of two networks $h_{1}$ and $h_{2}$, which mutually communicate useful information for boosting the final performance. Specifically, suppose we have a mini-batch of data $G=\{(x_{i},y_{i})\}$, where $y_{i}$ is the label with noise of the image $x_{i}$. $h_{1}$ and $h_{2}$ first separately predict the labels $\{\hat{y}_{i,h_{1}}\}$ and $\{\hat{y}_{i,h_{2}}\}$ of $\{x_{i}\}$, based on which $G$ is divided into $G_{s}=\{(x_{k},y_{k})\in G|\hat{y}_{k,h_{1}}=\hat{y}_{k,h_{2}}\}$ (instances with identical predictions) and $G_{d}=\{(x_{k},y_{k})\in G|\hat{y}_{k,h_{1}}\neq\hat{y}_{k,h_{2}}\}$ (instances with different predictions). Inspired by [29], we treat $G_{d}$ as “hard examples”, which can benefit the training of $h_{1}$ and $h_{2}$. To alleviate the negative effects caused by label noises in $G_{d}$, we explore “useful knowledge” in $G_{s}$ by selecting a small proportion of instances $G_{s_{1}}$ and $G_{s_{2}}$, according to the losses computed by $h_{1}$ and $h_{2}$, respectively. $G_{s_{i}}$ ($i\in\{1,2\}$) consists of instances that have the top $(1-d(T))$ smallest training losses by using $h_{i}$, and is concretely defined as:

$$\small\begin{matrix}G_{s_{i}}=\arg\min_{\tilde{G}_{s}\subset G_{s}:|\tilde{G}_{s}|\geq(1-d(T))|G_{s}|}\sum_{(x_{j},y_{j})\in\tilde{G}_{s}}\mathcal{L}_{h_{i}}(x_{j},y_{j}),\end{matrix}$$

(1)

where $\mathcal{L}_{h_{i}}(x_{j},y_{j})$ is the training loss of instance $x_{j}$ computed by $h_{i}$ ($i\in\{1,2\}$), and $|G_{s}|$ indicates the number of elements in $G_{s}$. Particularly,

$$\small\begin{matrix}d(T)=\xi\cdot\min\{\frac{T}{T_{k}},1\},\end{matrix}$$

(2)

is the drop rate that dynamically controls $|G_{s_{i}}|$, where $\xi$ is the maximum drop rate, and $T_{k}$ is the number of epochs after which $d(T)$ is no longer updated. The motivation lies in that we attempt to utilize more images for training at the beginning by setting $d(T)$ to a small value. Thereafter, $d(T)$ gradually increases, so that we only select correctly labeled instances with sufficiently high confidence, and drop noisy ones before the networks memorize them.

After obtaining $G_{d}$ and $G_{s_{i}}$ ($i\in\{1,2\}$), we treat $G_{d}\cup G_{s_{1}}$ as the “useful knowledge” to train $h_{2}$, provided by its peer network $h_{1}$. Similarly, $G_{d}\cup G_{s_{2}}$ is adopted to train $h_{1}$. The parameters $\theta_{h_{i}}$ of the network $h_{i}$ ($i\in\{1,2\}$) are updated by using the gradient $\nabla\mathcal{L}_{h_{i}}$ with a learning rate $\lambda$:

$$\begin{matrix}\theta_{h_{1}}\leftarrow\theta_{h_{1}}-\lambda\cdot\sum_{(x_{i},y_{i})\in G_{d}\cup G_{s_{2}}}\nabla\mathcal{L}_{h_{1}}(x_{i},y_{i}),\end{matrix}$$

(3)

$$\begin{matrix}\theta_{h_{2}}\leftarrow\theta_{h_{2}}-\lambda\cdot\sum_{(x_{i},y_{i})\in G_{d}\cup G_{s_{1}}}\nabla\mathcal{L}_{h_{2}}(x_{i},y_{i}).\end{matrix}$$

(4)

Either network in our method learns “useful knowledge” from its peer network, which can correct errors caused by instances in $G_{d}$ with label noises. Therefore, we denote our algorithm as “Peer-learning”. Through mutually communicating “useful knowledge”, both the performance of $h_{1}$ and $h_{2}$ can be improved.

Fine-grained module It is worth noting that the training strategy is independent of the backbone network structure $h_{1}$ and $h_{2}$. In the Peer-learning network, we realize fine-grained recognition by using existing fine-grained modules (e.g., B-CNN [59], NTS-Net [6], H-B-Pooling [7], DFL-CNN [8], OPAM [9], and Fast-MPN-COV [10]) as the backbone network structure for $h_{1}$ and $h_{2}$.

Why is our approach effective? Since instances in $G_{d}$ include biased predictions (at least one network gives a wrong prediction), updating the parameters only using instances in $G_{d}$ may result in errors and these errors will be directly transferred back to itself in the next mini-batch. These errors will be increasingly accumulated and previous works, like Decoupling [29] and MentorNet [55], cannot handle them explicitly. In our approach, as two networks have different learning abilities, they can communicate “useful knowledge” (instances with a small loss in $G_{s}$) with their peer network to update the parameters. Through these exchange procedures, errors induced by updating with $G_{d}$ can be identified and reduced by peer networks mutually. This is the reason why our approach is more robust to previous works like Decoupling and MentorNet. On the other side, compared to methods that update the model using only “clean” samples (e.g., Co-teaching [30]), the advantage of our approach is that we can also leverage the so-called “noisy” samples to promote model optimization. The motivation is that, in the group $G_{d}$, there are still many correctly classified instances. Thus, we can still learn “useful knowledge” from these instances to improve the model performance as long as the label noise is conquered.

Why do we need to update the neural networks with $G_{d}$? For each instance in $G_{d}$, at least one network in $h_{1}$ and $h_{2}$ has been given the wrong predictions. These wrong predictions give rise to an updated model that can promisingly result in better classification performance. However, updating neural networks $h_{1}$ and $h_{2}$ with $G_{d}$ also brings some errors. For example, the labels of instance $x$ predicted by $h_{1}$ and $h_{2}$ are $+1$ and $-1$, respectively. If the ground truth label of instance $x$ is $+1$, then $h_{2}$ treats $x$ as a wrong prediction and updating the parameters will promote the robustness of $h_{2}$. At the same time, $h_{1}$ correctly predicts the label of instance $x$, but wrongly regards it has been given an error prediction and so updates its parameters by treating $x$ as a wrong prediction will bring errors to $h_{1}$. What’s worse, these errors will be transferred back and increasingly accumulated.

Why can “useful knowledge” correct the errors induced by $G_{d}$? Memorization effects [24] indicate that, on noisy data sets, deep neural networks will first learn the clean and easy patterns in the initial epochs. Thus, they can filter out noisy instances through their loss values at the beginning of training. However, as the number of epochs increases, deep neural networks will eventually overfit on noisy labels. Our key idea is to drop these noisy labels before they are memorized. In our approach, we dynamically control the drop rate of noisy labels by the parameter $d(T)$. As the number of epoch $T$ increases, we gradually increase the drop rate $d(T)$, so that we can keep “clean” instances and drop noisy labels before the neural networks memorize them. Intuitively, small-loss instances in $G_{s}$ (easy examples) are more likely to be that correctly classified. On the other hand, networks $h_{1}$ and $h_{2}$ have different decision boundaries and thus have different abilities to learn. When training on noisy web images, they can have different abilities to filter out noisy labels. We cross-update neural networks using “useful knowledge” from each other. Through these exchange procedures of useful knowledge, different types of errors induced by $G_{d}$ can be mutually identified and reduced by peer networks. This process is similar to “peer-review”. When students check their own papers, it is very hard for them to find errors because they have some personal bias for the answers. Fortunately, they can ask peer classmates to review their papers.

In this work, we studied the problem of fine-grained recognition through noisy web images. To be specific, we first constructed two benchmark webly supervised fine-grained datasets WebFG-496 and WebiNat-5089. Then we proposed a novel method, termed Peer-learning, by simultaneously training two deep neural networks, both of which exploit and mutually communicate useful knowledge from noisy web images. Extensive experiments showed that our approach achieved state-of-the-art performance, compared to existing works. In the future, we plan to investigate the impact of class imbalance and design a more robust fine-grained model that takes class imbalance into account.

This work was supported by the National Natural Science Foundation of China (No. 61976116, 61772256, and 61772256), the Fundamental Research Funds for the Central Universities (No. 30920021135 and 30920041111), and the CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ-2020-022A).