Bridging the Gap between Classification and Localization
for Weakly Supervised Object Localization

To enhance the activation of the entire object region in CAM, we want the cosine similarity between $F_{u}$ and $\mathbf{w}_{c}$ to be high for $u$ belonging to the target object and low for the background region. Because high activation in $\mathcal{F}$ implies that there is a cue for classification at the corresponding location, we divide the region of the feature map into coarse foreground region $\mathcal{R}^{\text{norm}}_{\text{fg}}$ and background region $\mathcal{R}^{\text{norm}}_{\text{bg}}$ based on a normalized $\mathcal{F}$.

$\tau_{\text{fg}}$ and $\tau_{\text{bg}}$ are constant thresholds that determine the foreground and background regions, respectively. Note that $\tau_{\text{fg}}$ and $\tau_{\text{bg}}$ are not the same; therefore, there is an unknown region that is not included in either $\mathcal{R}^{\text{norm}}_{\text{fg}}$ or $\mathcal{R}^{\text{norm}}_{\text{bg}}$. To increase $\mathcal{S}_{u}$ in $\mathcal{R}^{\text{norm}}_{\text{fg}}$ and suppress it in $\mathcal{R}^{\text{norm}}_{\text{bg}}$, we define the similarity loss as follows:

There still remains a possibility that some parts of the object region have low activation in $\hat{\mathcal{F}}$. In this case, $\mathcal{L}_{\text{sim}}$ may not be sufficient for the alignment. Therefore, we introduce an additional loss term to increase $\hat{\mathcal{F}}$ in every candidate region belonging to the target object. Because a positive $\mathcal{S}_{u}$ indicates that $u$ is making a positive contribution to increasing the classification logit, the regions with positive similarity can be treated as candidates for the object region. Therefore, we force this area to be activated. We estimate the object region, $\mathcal{R}^{\text{sim}}_{\text{fg}}$, and background region, $\mathcal{R}^{\text{sim}}_{\text{bg}}$, based on $\mathcal{S}_{u}$ as

With each estimated region, we define the norm loss in a manner similar to Eq. 6, as follows:

For fine-grained classification, such as bird species classification, the object to be recognized is the same across classes. In this case, we define the region with a non-positive similarity with any class as $\mathcal{R}^{\text{sim}}_{\text{bg}}$ and the other as $\mathcal{R}^{\text{sim}}_{\text{fg}}$. In general, the regions $\mathcal{R}^{\text{sim}}_{\text{bg}}$ and $\mathcal{R}^{\text{sim}}_{\text{fg}}$ are defined with a similarity with a target class.

The two loss terms $\mathcal{L}_{\text{sim}}$ and $\mathcal{L}_{\text{norm}}$ operate complementary. Through the minimization of $\mathcal{L}_{\text{sim}}$, the value of $\mathcal{S}$ in the region that is highly activated in $\hat{\mathcal{F}}$ increases. Through the minimization of $\mathcal{L}_{\text{norm}}$, the value of $\hat{\mathcal{F}}$ in the region with high similarity increases. After the joint minimization of $\mathcal{L}_{\text{sim}}$ and $\mathcal{L}_{\text{norm}}$, the activated region in $\hat{\mathcal{F}}$ and that in $\mathcal{S}$ become similar.

We can expect the successful alignment by $\mathcal{L}_{\text{sim}}$ when the estimation of $\mathcal{R}^{\text{norm}}_{\text{fg}}$ and $\mathcal{R}^{\text{norm}}_{\text{bg}}$ is accurate: $\hat{\mathcal{F}}$ is consistently large over the entire object region and small over the background region.

Because the value of $\mathcal{F}$ at the most discriminative region is significantly larger than that at the other region, the value of the normalized map $\hat{\mathcal{F}}$ at the less discriminative part but belonging to the object region becomes small.

We introduce consistency with attentive dropout, a method to distribute the activation to the target object region. We adopt $L_{1}$ loss between the two feature maps $F$ and $F_{\text{drop}}$: $F$ is the feedforward result of an intermediate feature map $F^{\prime}$, and $F_{\text{drop}}$ is the feedforward result of $F^{\prime}_{\text{drop}}$ obtained by intentionally dropping large activations from $F^{\prime}$. Fig. 4 shows the overall process of obtaining $F^{\prime}_{\text{drop}}$ for consistency with attentive dropout. In $F^{\prime}$, the activation at the spatial location whose channel-wise averaged activation is larger than $\gamma$ is dropped with probability $p$. The stochastic dropout prevents all information in the highly activated area from being eliminated. The loss for consistency with attentive dropout is as follows:

There have been several attempts that utilize a similar erasing mechanism [16, 5, 32]. They train a classifier to preserve the predicted labels before and after erasing highly activated features. In contrast, our method explicitly regularizes a model to yield a similar feature map even after the highly activated features are dropped. This decreases the dependency on the dropped features, resulting in more evenly distributed activation compared to the other methods.

With cross-entropy loss for classification, $\mathcal{L}_{\text{CE}}$, the total cost function is defined as follows:

where $\lambda_{\text{drop}}$, $\lambda_{\text{sim}}$, and $\lambda_{\text{norm}}$ are hyperparameters for balancing the losses. The feature direction alignment is better applied after training the classifier to some extent to obtain a suitable feature map for classification. Thus, for the first few epochs (i.e., the warm stage), we train a model only with $\mathcal{L}_{\text{CE}}$ and $\mathcal{L}_{\text{drop}}$:

We compare our method to the recent WSOL methods. For other WSOL methods, we report the localization performance of the original papers or that reproduced by [4, 10, 1, 24]¹¹1https://github.com/clovaai/wsolevaluation. Our method consistently outperforms existing WSOL methods using a single branch, across the datasets and the backbones by a large margin.

Tab. 1 shows the localization performance on the CUB-200-2011 [27] test set, using VGG16 as a backbone. Our method achieves an 11.87%p improvement in Top-1 Loc and a 16.87%p improvement in GT Loc over the work of Bae et al. [1], which is the state-of-the-art method among the CAM-based methods. Furthermore, our method outperforms the methods adopting an additional branch for localization. Our method improves Top-1 Loc by 1.57%p and GT-Loc by 3.91%p improvement in GT Loc compared to FAM [17].

Tab. 2 shows the results using ResNet50 as a backbone. It shows that our method consistently outperforms the existing methods by a large margin ($>$13%p), using a different backbone. Tab. 3 shows the localization performance on the ImageNet-1K [21] validation set, based on VGG16 and ResNet50. Our method achieves the state-of-the-art performance in the ImageNet-1K dataset regardless of the backbone, and only Top-1 Loc with ResNet50 is the second best after I²C with a marginal difference.

Additionally, we compare our MaxBoxAccV2 [4] scores with other state-of-the-art methods on the CUB-200-2011 and ImageNet-1K in Tab. 4. It shows that our method outperforms the most recent methods by a large margin for all IoU thresholds with various backbones and datasets. Especially, our method improves the score with IoU threshold of 0.7, which is strict accuracy, by 21.0%p and 17.4%p with VGG16 and ResNet50 on the CUB-200-2011 dataset, respectively, compared with the work of Ki et al. [9].

Fig. 5 shows some examples of localization results from the vanilla method [35] and from our method on the CUB-200-2011 and ImageNet-1K datasets. It shows that the model trained with our method captures the target object region more accurately than the vanilla model. On the CUB-200-2011 dataset, while the vanilla model fails to identify the tails, legs, and wings of birds, the classifier trained with our method successfully identifies them.

Fig. 7(a) shows the distribution of $\mathcal{S}_{u}$ inside the ground truth bounding boxes from the vanilla method and our method. Note that the bounding boxes include not only the target object but also the background region. As the training progresses with our method, the similarity gradually splits into negative and large positive values. This shows that our method effectively increases the similarity for the foreground region and decreases it for the background region. In contrast, for the vanilla method, the similarity is clustered in small positive values, making no distinction between the two.

We perform a series of ablation studies on the CUB-200-2011 dataset using VGG16 as the backbone.

For the balancing factors in loss, we find the best localization performance at 0.5 for $\lambda_{\text{sim}}$, 0.15 for $\lambda_{\text{norm}}$, and 3 for $\lambda_{\text{drop}}$, respectively. As shown in Fig. 8(a), the localization performance is most sensitively affected by $\lambda_{\text{sim}}$. $\lambda_{\text{norm}}$ insignificantly changes the performance. The performance tends to decrease when the constraint with $\lambda_{\text{drop}}$ becomes too strong as 4.

For the hyperparameters of the feature direction alignment, we set $\tau_{\text{fg}}$ and $\tau_{\text{bg}}$ for $\mathcal{L}_{\text{sim}}$ to 0.6 and 0.1, respectively. They determine the coarse foreground and background regions. Fig. 8(b) shows that varying those thresholds has little effect on the performance. The hyperparameters $\gamma$ and $p$ determine the drop of the activation in the intermediate feature map. $\gamma$ and $p$ for $\mathcal{L}_{\text{drop}}$ are set to 0.8 and 0.5, respectively. When $\gamma$ is moderately large between 0.7 and 0.9, there is no significant change in the performance, but when $\gamma$ is too low, i.e., 0.6, the performance decreases. From the results with various $p$, we observe that stochastic dropout produces little change of GT Loc regardless of the drop probability, but deterministic dropout with a probability of 1.0 yields a significant drop in the localization performance. This indicates that less but sufficient discriminative information should be maintained for a good localization performance.

A threshold is required to draw a bounding box around an object from a continuous localization map. Fig. A1 shows that the change of localization accuracy with various IoUs when varying the threshold. In each plot with IoU $\delta$, the maximum value becomes MaxBoxAccV2($\delta$) score. For all $\delta$, the curve of our method is consistently above the curve of the vanilla method, which shows that the superiority of the localization performance of our method does not depend on the threshold. When $\delta$ is 0.3 and 0.5, the curve of our method nearby a maximum value is flatter than the curve of the vanilla method. This shows that our method is less sensitive to the threshold for a bounding box than the vanilla method. When $\delta$ is 0.7, our method is sharper than the vanilla method, but the localization accuracy of our method is more than twice that of the vanilla method, so the flatness comparison is meaningless.

Fig. A3 shows some examples of CAM, $\mathcal{F}$, and $\mathcal{S}$ from the vanilla method and our method on the CUB-200-2011 and ImageNet-1K datasets. In $\mathcal{S}$ from the vanilla method, the overall values are similar and some regions that belong to the object have low values. For instance, in the ‘car’ example (in the second row and second column of the ImageNet-1K dataset), the middle part of the object has low similarity, resulting in low activation in the CAM. Different from the vanilla method, the values of $\mathcal{S}$ from our method are high in the object regions and low in the background regions. Furthermore, the values of $\mathcal{F}$ are high across the entire object region. This makes the CAM that captures more object region.

We compare consistency with attentive dropout with EIL [16], the most recent one among the previous erasing methods [5, 16, 32], on the CUB-200-2011 using VGG16. As mentioned in the main paper, attentive dropout directly regularizes feature activation, whereas EIL indirectly influences the activation through class prediction. Fig. A2 shows the different effect on feature activation of consistency with attentive dropout and EIL. To encourage a model to predict correct class without highly activated region, EIL enhances the maximum value. In contrast, consistency with attentive dropout reduces the maximum value through regularization. As a result, attentive dropout distributes the activations more effectively than EIL (Fig. 7(b) in the main paper).

Fig. A4 compares the localization results from the vanilla method [35] and our method on the CUB-200-2011 and ImageNet-1K datasets. While the vanilla method misses the less discriminative parts, e.g., wings and tails of birds and bodies of animals, our method successfully captures the entire object region.

In the main paper, we mention as a limitation that there are several hyperparameters to be decided in our method. We provide further analysis with a different dataset and backbone than that presented in the main paper.