Fine-Grained Attention for Weakly Supervised Object Localization

Let $\mathbf{X}\in\mathbb{R}^{C\times H\times W}$ be an input feature tensor, where $C$, $H$, and $W$ denote the dimension of the channel, height, and width, respectively. To condense the global distribution of an input feature tensor $\mathbf{X}$ in the triple views, we apply an average pooling in each dimension of the tensor, i.e., channel, height, and width as follows:

	$$\displaystyle\mathbf{c}=\text{AvgPool}_{\text{wh}}(\mathbf{X})$$		(1)
	$$\displaystyle\mathbf{h}=\text{AvgPool}_{\text{w}}(\mathbf{X})$$		(2)
	$$\displaystyle\mathbf{w}=\text{AvgPool}_{\text{h}}(\mathbf{X})$$		(3)

where $\text{AvgPool}_{\{\cdot\}}$ is an average pooling operator with respect to the dimensions of $\{\cdot\}$.

After computations in Eq. (1)-(3), the three pooled features of $\mathbf{c}\in\mathbb{R}^{C\times 1\times 1}$, $\mathbf{h}\in\mathbb{R}^{C\times H\times 1}$, and $\mathbf{w}\in\mathbb{R}^{C\times 1\times W}$ can be regarded as a summary of the extracted features in $\mathbf{X}$ from different viewpoints. Surely, the three views carry different information distributed in the input feature tensor $\mathbf{X}$. That is, $\mathbf{c}$ captures which feature representations are highly activated, and $\mathbf{h}$ and $\mathbf{w}$ reflect, respectively, where the discriminative features distributed vertically and horizontally across channels.

Subsequently, in order to utilize their local interaction among units in each pooled feature [27], we apply a 1D convolution with a kernel size of $k$ and zero-padding without biases, thus keeping their dimensionality. Then, a batch normalization [14] and a non-linear activation function are applied as follows:

	$$\displaystyle\mathbf{z}_{\text{h}}=\sigma(\text{BN}(\mathbf{W}_{\text{h}}(\mathbf{h})))$$		(4)
	$$\displaystyle\mathbf{z}_{\text{w}}=\sigma(\text{BN}(\mathbf{W}_{\text{w}}(\mathbf{w})))$$		(5)
	$$\displaystyle\mathbf{z}_{\text{c}}=\sigma(\text{BN}(\mathbf{W}_{\text{c}}(\mathbf{c})))$$		(6)

where $\sigma(\cdot)$ is a sigmoid function and $\mathbf{W}_{\{\cdot\}}$ indicates the 1D convolutional layer for the respective pooled features. Here, $\mathbf{z}_{\text{h}}\in\mathbb{R}^{C\times H\times 1}$, $\mathbf{z}_{\text{w}}\in\mathbb{R}^{C\times 1\times W}$, and $\mathbf{z}_{\text{c}}\in\mathbb{R}^{C\times 1\times 1}$ corresponds to the resulting triple-view attentions.

We propose to expand the triple-view attentions of $\mathbf{z}_{\text{h}}$, $\mathbf{z}_{\text{w}}$, and $\mathbf{z}_{\text{c}}$ in the size of an input feature tensor $\mathbf{X}$, by which it is beneficial to reflect the attention information back into the input feature tensor $\mathbf{X}$ in a fine-grained manner. Therefore, we create an attention map $\mathbf{M}\in\mathbb{R}^{C\times H\times W}$ of the same size of the input feature map $\mathbf{X}$ by means of an expansion function $f$ defined by an outer sum as follows:

$$\mathbf{M}=\sigma(f(\mathbf{z}_{\text{h}},\mathbf{z}_{\text{w}},\mathbf{z}_{\text{c}})).$$

(7)

It should be note that values in the attention map $\mathbf{M}$ are likely to be different from each other, resulting in a fine-grained attention map. Our fine-grained attention map representation method is of major contrast to the previous attention-based methods that learn a coarse attention map, having the same values across elements within the same channel, for example.

With the attention tensor estimated in Eq. (7), we then apply it to the input feature tensor. In particular, we consider computational approaches as follows:

$\displaystyle\mathbf{\hat{X}}=\left\{\begin{array}[]{ll}\mathbf{X}\odot\mathbf{M}&\text{(non-residual)}\\ \mathbf{X}\oplus(\mathbf{X}\odot\mathbf{M})&\text{(residual)}\end{array}\right.$

(10)

where $\odot$ and $\oplus$ denote Hadamard product and element-wise summation, respectively.

Fundamentally, although those two approaches employ fine-level attention maps, enabling detailed feature calibration at element-level units, they work in different ways in terms of feature representation learning. The non-residual approach is to mine and calibrate as many discriminative features as possible by multiplying the input feature tensor with the corresponding attention tensor. Meanwhile, in the residual approach, because of the element-wise sum operation (i.e., a residual operation), the element-wise multiplication part between the input feature tensor and the attention tensor is likely to excite the locations where the input feature tensor $\mathbf{X}$ may have less activations. Hence, the attention module described in Section 3.2 is trained to give more attention to locations where the input feature tensor presents relatively small activations. Accordingly, the output attention tensor in $\mathbf{M}$ plays the role of exciting the less activated regions in $\mathbf{X}$ while inhibiting the more activated regions. This interpretable phenomenon is clearly observed from our experimental results in Fig. 5 and Fig. 6.

Datasets. We validated our RFGA over three public datasets for WSOL, i.e., CUB-200-2011 [25], FGVC Aircraft [22], and Stanford Dogs [15]. First, CUB-200-2011 includes a total of $11,788$ images from $200$ bird categories, which is divided into $5,994$ images for training and $5,794$ images for evaluation. FGVC Aircraft consists of $10,000$ images across $100$ aircraft categories with $3,334$ for training, $3,333$ for a validation, and $3,333$ for testing. Stanford Dogs contains a total of $20,580$ dog samples in $102$ categories, which is composed of $12,000$ training samples and $8,580$ test samples.

Competing methods. We compared our RFGA with five existing state-of-the-art WSOL methods; e.g., CAM [40], HaS [24], ACoL [36], ADL [5], and CutMix [35]. Further, in order to see the effectiveness of attention methods in WSOL, we compared to three other context fusion based attention methods; e.g., SENet [12], CBAM [31], and ECA-Net [27] as well.

Evaluation metric. In order for quantitative evaluation, we used MaxBoxAcc [4] over the IoU thresholds $\delta\in\{0.5,0.6,0.7,0.8,0.9\}$ at the optimal activation map threshold. A threshold of activation map, $\tau$, is set between 0 and 1 at 0.01 intervals. Therefore, we finally obtained the results, measuring various localization performances over threshold $\tau$ for an activation map at various levels of $\delta$.

We used ResNet-50 [10] pre-trained with ImageNet data as a backbone network. In order to obtain localization maps, we used feature maps of $1\times 1$ convolutional layers, similar to ACoL [36]. For the kernel size $k$ in the triple-view attentions, we used $3$, according to the work of [27]. The input images of training were resized to $600\times 600$ and then we cropped randomly $448\times 448$ patches from the resized images. In addition, the input images were flipped horizontally with a probability of $0.5$. Meanwhile, the test images were resized to $448\times 448$. We trained our RFGA for a total of $60$ epochs with a mini-batch size of $20$ and an initial learning rate of $0.01$ that was decreased by $0.1$ after every $15$ epochs. Further, we used the stochastic gradient descent optimizer with a momentum of $0.9$. More details for the settings of comparative methods can be found in Supplemantary. We implemented all methods in PyTorch¹¹1https://pytorch.org and trained with Titan X GPU.

Quantitative evaluation. Table 1 summarizes the performance of the competing methods at the optimal activation map threshold $\tau$. We observed that our RFGA method outperformed localization performance compared to other competing methods in terms of MaxBoxAcc and Mean Intersection over Union (mIoU) which is the average IoU of all images at optiaml $\tau$. Further, it is noteworthy that our RFGA showed its superiority to all competing methods in all cases including various IoU thresholds over three datasets.

Qualitative visualization. We visualized the predicted localization bounding boxes and activation maps for all methods in Fig. 3. Each image presents the localization results at the optimal threshold $\tau$ where the IoU of the bounding box from the activation map achieves maximum value. We observed that RFGA elaborately localized the entire part of an object for CUB-200-2011, FGVC aircraft, and Stanford Dogs datasets. While most competing methods focused on the partial objects or covered in excess of the exact object region, our RFGA tightly bounded the entire and specified region of the object in an image, thereby achieving the best localization performance.

Classification. We additionally report the classification accuracy in Table 2 to explore the relation between localization and classification. By following to work of [3], we selected the model at the last epoch without regard to validation results. However, we observed that most WSOL methods showed a tendency of achieving the best localization performance in the early stage of training in spite of low classification performance. Consequently, we believe that the classification performance is not correlated with the localization performance, consistent with the work in [4].

Hyperparameter analysis. We plotted the change of localization performance (MaxBoxAcc [4]) of all methods by varying the value of $\tau$ in Fig. 4. Our proposed RFGA showed the best performance at a smaller $\tau$ value than that of other comparative methods. From the results, we could infer that most of the high activation values in our RFGA-based feature tensor were well aligned to the object-related region.

Visualization of attention maps. In order to get an insight into the working of our RFGA, we visualized all triple-view attention maps (i.e., $\mathbf{z}_{c}$, $\mathbf{z}_{w}$, $\mathbf{z}_{h}$) as well as the final combined attention map (i.e., $\mathbf{M}$) (top) and the input feature map $\mathbf{X}$, the element-wise product of $\mathbf{X}$ and $\mathbf{M}$ (i.e., $A(\mathbf{X})$), the resulting output feature map $\hat{\mathbf{X}}$ via a residual approach, and the difference between $\mathbf{X}$ and $\hat{\mathbf{X}}$ (bottom) in Fig. 5. We took an average of each attention vector along the channel axis and expanded the averaged vectors, $\mathbf{z}_{c}$, $\mathbf{z}_{w}$, and $\mathbf{z}_{h}$ to a matrix by repetition for a visualization. It should be noted that we normalized each matrix in a range of $[0,1]$. Contrary to the activation map of CAM that only focuses on the body of a bird, our RFGA pays additional attention to the wings, resulting in the entire object attention. In accordance with Fig. 3, we validated the effectiveness of our fine-grained calibration of features in WSOL.

Effect of triple-view attention. We assumed that our RFGA could localize an object from a variety of fine-grained information. To validate the effectiveness of the fine-grained attention, we conducted ablation studies with respect to each of the attention maps generated from different views, i.e., channel $\mathbf{z}_{c}$, vertical $\mathbf{z}_{h}$, and horizontal $\mathbf{z}_{w}$. We employed only one-view attention out of those three view when training the same architecture, and reported the results in Table 3. Our triple-view attentions method clearly outperformed all the ablation cases. Based on those results, we believe that our fine-grained attention map from the triple-view attentions is capable of calibrating features through the complementary relations inherent in the input feature tensor.

Effect of residual connection. In order to investigate residual connection effect, we compared the residual approach and the non-residual approach in Eq. (10) in terms of the localization task in Table 1 and the classification task in Table 2. We also visualized their respective attention maps in Fig. 6. The residual approach generated the attention maps that focused on both the most and the less discriminative regions of an object. Meanwhile, for the non-residual approach, their attention maps showed the opposite patterns to those of the residual-based maps. Based on the understanding of a residual operation, note that $\mathbf{X}\oplus A(\mathbf{X})$ leads the function $A(\mathbf{X})$ to learn some amount of information that the input feature tensor $\mathbf{X}$ may have missed or less emphasized. From the viewpoint of attention map generation, the role of $A(\mathbf{X})$ can be interpreted as to inhibit the regions of high activation values (as those are already well represented in $\mathbf{X}$) and to excite the less activated regions where the target task-related information is inherent. Here, the inhibition effect can be related to those of the specially-designed module in ACoL[36] and ADL[5] that erase the discriminative features.