Informative Class Activation Maps

Contemporary classification convolutional neural networks (CNNs) consist of stacks of convolutional layers interleaved with pooling layers for extracting visual features. These convolutional layers result in feature maps. A feature map is a collection of 2-dimensional grids. The size of the feature map depends on the structure of convolution and pooling layers. Generally the feature map is smaller than the original image. The number of feature maps corresponds to the number of convolutional filters. The feature maps from the final convolutional layer are usually averaged, flattened and fed into the fully-connected layer for classification (Lin et al., 2014). Given $K$ feature maps $g_{1},..,g_{K}$, the fully-connected layer consists of weight matrix $W\in\mathbb{R}^{M\times K}$, where $w_{k}^{y}$ represents the scalar weight corresponding to class $y$ for feature $k$. We use $g_{k}(a,b)$ to denote a value of 2-dimensional spatial point $(a,b)$ with feature $k$ in map $g_{k}$. In (Choe & Shim, 2019), the authors propose a way to interpret the importance of each point in feature maps. The importance of spatial point $(a,b)$ for class $y$ is defined as a weighted sum over features:

$\displaystyle M_{y}(a,b)=\sum_{k}w_{k}^{y}g_{k}(a,b).\vspace{-4mm}$

(1)

We redefine $M_{y}(a,b)$ as an intensity of the point $(a,b)$. The collection of these intensity values over all grid points forms a class activation map (CAM). CAM highlights the most relevant region in the feature space for classifying $y$. The input going to the softmax layer corresponding to the class label $y$ is:

$\displaystyle\sum_{a,b}M_{y}(a,b)=n(\mathbf{x})_{y}.\vspace{-4mm}$

(2)

Intuitively, weight $w_{k}^{y}$ indicates the overall importance of the $k$th feature to class $y$, and intensity $M_{y}(a,b)$ implies the importance of the feature map at spatial location $(a,b)$ leading to the classification of image $\mathbf{x}$ to $y$.

WSOL: The aim of WSOL is to identify the region containing the target object in an image given a label, without any pixel-level supervision. Previous approaches tackle WSOL by creating a bounding box from the CAM (Choe & Shim, 2019). Such a CAM contains all important locations that exceed a certain intensity threshold. The box is then upsampled to match the size of the original image.

In (Qin et al., 2019), the authors show that softmax classifier carries an explicit implication between inputs and labels in terms of information theory. We extend the notion of mutual information from being a pair of an input image and a label to regions of the input image and labels to capture the regions that have high mutual information with labels.

To simplify the discussion, we assume here that there is only one feature map, i.e., $K=1$. However, the following results can be easily applied to the general cases where $K>1$ without loss of generality. We introduce a region $R$ containing a subset of grid points in feature map $g$.

Mutual information is an expectation of the point-wise mutual information (PMI) between two variables, i.e., $\mathbb{I}(\mathbf{X},Y)=\mathbb{E}_{\mathbf{x},y}[\text{PMI}(\mathbf{x},y)]$. Given two instances of variables, we can estimate their PMI of $\mathbf{x}$ and $y$ as:

$\displaystyle\operatorname{PMI}(\mathbf{x},y)=n(\mathbf{x})_{y}-\log\sum_{y^{\prime}=1}^{M}\exp(n(\mathbf{x})_{y^{\prime}})+\log M.$

The PMI is close to $\log M$ if $y$ is the maximum argument in log-sum-exp. To find a region which is the most beneficial to the classification, we compute the difference between PMI with true label and the average of the other labels and decompose it into a point-wise summation as

	$\displaystyle\operatorname{Diff}(\operatorname{PMI}(\mathbf{x}))=\operatorname{PMI}(\mathbf{x},y^{*})-\frac{1}{M-1}\sum_{y^{\prime}\neq y^{*}}\operatorname{PMI}(\mathbf{x},y^{\prime})$
	$\displaystyle=\sum_{(a,b)\in g}w^{y*}g(a,b)-\frac{1}{M-1}\sum_{y^{\prime}\neq y^{*}}w^{y^{\prime}}g(a,b).$

The point-wise decomposition suggests that we can compute the PMI differences with respect to a certain region. Based on this observation, we propose a new CAM, named informative CAM or infoCAM, with the new intensity function $M_{y}^{\operatorname{Diff}}(R)$ between region $R$ and label $y$ defined as follows:

$\displaystyle M_{y}^{\operatorname{Diff}}(R)=\sum_{(a,b)\in R}w^{y}g(a,b)-\frac{1}{M-1}\sum_{y^{\prime}\neq y}w^{y^{\prime}}g(a,b).\vspace{-2mm}$

The infoCAM highlights the region which decides the classification boundary against the other labels. Moreover, we further simplify the above equation to be the difference between PMI with the true and the most-unlikely labels according to the classifier’s outputs, denoting as infoCAM+, with the new intensity:

$\displaystyle M_{y}^{\operatorname{Diff}^{+}}(R)=\sum_{(a,b)\in R}w^{y}g(a,b)-w^{y^{\prime}}g(a,b),$

(3)

where $y^{\prime}=\underset{m}{\arg\min}\sum_{(a,b)\in R}w^{m}g(a,b)$.

The complete procedure of WSOL with infoCAM is visually illustrated in Figure 1.

We evaluate WSOL performance on CUB-200-2011 (Wah et al., 2011) and Tiny-ImageNet (Fei-Fei, ). CUB-200-2011 consists of 200 bird specifies. Since the dataset only depicts birds, not including other kinds of objects, variations due to class difference are subtle (Dubey et al., 2018). Such nuance-only detection can lead to localisation accuracy degradation (Choe & Shim, 2019).

Tiny-ImageNet is a reduced version of ImageNet. Compared with the full ImageNet, training classifiers on Tiny-ImageNet is faster due to image resolution reduction and quantity shrinkage, yet classification becomes more challenging (Odena et al., 2017).

To perform an evaluation on localisation, we first need to generate a bounding box for the object within an image. We generate a bounding box in the same way as in (Zhou et al., 2016). That is, after evaluating infoCAM within each region of an image, we only retain the regions whose infoCAM values are more than 20% of the maximum infoCAM and abandon all the other regions. Then, we draw the smallest bounding box that covers the largest connected component.

We follow the same evaluation metrics in (Choe & Shim, 2019) to evaluate localisation performance with two accuracy measures: 1) localisation accuracy with known ground truth class (GT Loc.), and 2) top-1 localisation accuracy (Top-1 Loc.). GT Loc. draws the bounding box from the ground truth of image labels, whereas Top-1 Loc. draws the bounding box from the predicted most likely image label and also requires correct classification. The localisation of an image is judged to be correct when the intersection over union of the estimated bounding box and the ground-truth bounding box is greater than 50%.

We adopt the same network architectures and hyper-parameters as in (Choe & Shim, 2019), which shows the current state-of-the-art performance. Specifically, the network backbone is ResNet50 (He et al., 2016) and a variation of VGG16 (Szegedy et al., 2015), in which the fully connected layers are replaced with global average pooling (GAP) layers to reduce the number of parameters. The traditional softmax is used as the final layer since both datasets are well balanced. InfoCAM requires the region parameter $R$. We apply a square region for the region parameter $R$. The size of the region $R$ is set as $5$ and $4$ for VGG and ResNet in CUB-200-2011, respectively, and $3$ for both VGG and ResNet in Tiny-ImageNet.

These models are tested with the Attention-based Dropout Layer (ADL) to tackle the localisation degradation problem (Choe & Shim, 2019). ADL is designed to randomly abandon some of the most discriminative image regions during training to ensure CNN-based classifiers cover the entire object. The ADL-based approaches demonstrate state-of-the-art performance in CUB-200-2011 (Choe & Shim, 2019) and Tiny-ImageNet (Choe et al., 2018) for the WSOL task and are computationally efficient. We test ADL with infoCAMs to enhance WSOL capability.

To prevent overfitting in the test dataset, we evenly split the original validation images to two data piles, one still used for validation during training and the other acting as the final test dataset. We pick the trained model from the epoch that demonstrates the highest top-1 classification accuracy in the validation dataset and report the experimental results with the test dataset. All experiments are run on two Nvidia 2080-Ti GPUs, with the PyTorch deep learning framework (Paszke et al., 2017).

Table 1 shows the localisation results on CUB-200-2011 and Tiny-ImageNet. The results demonstrate that infoCAM can consistently improve accuracy over the original CAM for WSOL under a wide range of networks and datasets. Both infoCAM and infoCAM+ perform comparably to each other. ADL improves the performance of both models with CUB-200-2011 datasets, but it reduces the performance with Tiny-ImageNet. We conjecture that dropping any part of a Tiny-ImageNet image with ADL significantly influences classification since the images are relatively small.

Figure 2 highlights the difference between CAM and infoCAM. The figure suggests that infoCAM gives relatively high intensity on the object to compare with that of CAM, which only focuses on the head part of the bird. Figure 5 in the Appendix presents additional examples of visualisation for comparing localisation performance of CAM to infoCAM, both without the assistance of ADL . From these visualisations, we notice that the bounding boxes generated from infoCAM are formed closer to the objects than the original CAM. That is, infoCAM tends to precisely cover the areas where objects exist, with almost no extraneous or lacking areas. For example, CAM highlights the bird heads, whereas infoCAM also covers the bird bodies.

So far, we have shown the results of localisation from a multi-class classification problem. We further extend our experiments on localisation to multi-label classification problems. A softmax function is a generalisation of its binary case, a sigmoid function. Therefore, we can apply infoCAM to each label for a multi-label classification problem, which is a collection of binary classification tasks.

For the experiment, we construct a double-digit MNIST dataset where each image contains up to two digits randomly sampled from the original MNIST dataset (LeCun et al., 2010). We locate one digit on the left-side, and the other on the right-side. Some of the images only contain a single digit. For each side, we first decide whether to include a digit from a Bernoulli distribution with mean of 0.7. Then each digit is randomly sampled from a uniform distribution. However, we remove the images that contain no digits. Random samples from the double-digit MNIST are shown in 4(a).

We first compare the classification accuracy results between using the original sigmoid and PC-sigmoid. As shown in Table 2, PC-sigmoid increases the classification accuracy for each digit type on the test set. InfoCAM improves the localisation accuracy for the WSOL task as well. CAM achieves the localisation accuracy of 91%. InfoCAM enhances the localisation accuracy to 98%. Qualitative visualizations are displayed in Figure 4. We aim to preserve the regions of an image that are most relevant to a digit, and erase all the other regions. From the visualization, one can see that infoCAM localizes digits more accurately than CAM.