Learning from Counting: Leveraging Temporal Classification for Weakly Supervised Object Localization and Detection

Existing efforts to improve the WSOD performance depends mainly on two research directions - developing better proposal classifiers and developing new RPNs to generate more accurate proposals. [Bilen and Vedaldi(2016)] developed an effective, end-to-end deep network for WSOD, in which a pre-trained CNN is used for feature extraction and two data streams are developed to undertake detection and recognition in parallel. However, this model tends to assign a higher score to a proposal that contains the most discriminative part of an object rather than the entire object. [Tang et al.(2017)Tang, Wang, Bai, and Liu] designed a strategy to assign the same image label for proposals that have significant overlaps with those receiving high scores during the weak supervision phase. An Online Instance Classifier Refinement (OICR) algorithm was then developed to use these proposals as pseudo ground-truths to classify the training images. Through continuous refinements, the proposed WSOD can achieve better instance recognition than the network in [Bilen and Vedaldi(2016)]. [Wan et al.(2019)Wan, Liu, Ke, Ji, Jiao, and Ye] developed a C-MIL (Continuation multiple instance learning) model to achieve WSOD using new loss functions. [Gao et al.(2018)Gao, Li, Yu, Morariu, and Davis] developed a Count-guided Weakly Supervised Localization (C-WSL) network to achieve high-confidence OD. This work addressed the issue of the tendency to draw a proposal containing multiple objects in existing weakly supervised detectors. [Gao et al.(2019)Gao, Liu, Guo, Ye, Wan, You, and Fan] leveraged segmentation maps with coupled multiple instance detection network (C-MIDN) to refine the proposals before sending them to the classifier, which also uses the OICR model. [Yang et al.(2019)Yang, Li, and Dou] integrated bounding-box regression (REG) into MIL as a single end-to-end network and enhanced the original feature map with implicit object location information by attention maps from images (guided attention module) (GAM). Two MIL approaches were adopted in this work: OICR and Proposal Cluster Learning (PCL)[Tang et al.(2018a)Tang, Wang, Bai, Shen, Bai, Liu, and Yuille].

All the above approaches aim to improve one or more stages of a WSOD network. However, their proposal generation processes mostly rely on mature techniques, such as SS or EB. However, [Hosang et al.(2015)Hosang, Benenson, Dollár, and Schiele] and [Tang et al.(2018b)Tang, Wang, Wang, Yan, Liu, Huang, and Yuille] argue that the quality of proposals has a significant impact on the overall OD performance. Therefore, in recent years, more studies have been undertaken to improve proposal generation in RPNs. [Diba et al.(2017)Diba, Sharma, Pazandeh, Pirsiavash, and Van Gool] developed cascaded multiple networks with created class activation maps to infer better region proposals. [Tang et al.(2018b)Tang, Wang, Wang, Yan, Liu, Huang, and Yuille] proposed a two-stage network to improve the quality of generated proposals. The refined proposals are then sent to another WSOD [Tang et al.(2017)Tang, Wang, Bai, and Liu] to perform classification. Our research is also towards developing a RPN which can generate better proposals. But we take a very different approach - instead of relying on traditional object detection in a 2D realm, our approach converts the 2D object detection problem into a 1D sequence data segmentation problem and solves it by a novel use of LSTM and a count-based CTC.

LSTM [Hochreiter and Schmidhuber(1997)] is a type of RNN that was originally designed to model sequence data. LSTM can propagate information through lateral connections to model short- and long-term context dependencies. Regarding its usage in image processing tasks [Byeon et al.(2015)Byeon, Breuel, Raue, and Liwicki, Bell et al.(2016)Bell, Lawrence Zitnick, Bala, and Girshick], LSTM networks need to be extended from the temporal domain to the spatial domain and from single-dimensional learning to multi-dimensional learning [Graves and Schmidhuber(2009)]. [Visin et al.(2015)Visin, Kastner, Cho, Matteucci, Courville, and Bengio] coupled four uni-dimensional RNNs to sweep across an image in four directions to replace the common convolutional-pooling layer. The network ensures that the output activation will appear in a specific location with respect to the whole image rather than a local context window of a CNN. In our work, we adopt the LSTM structure to identify critical points on a target object by taking advantage of its power in capturing global contexts and context dependencies in serialized data.

CTC [Graves et al.(2006)Graves, Fernández, Gomez, and Schmidhuber] is a type of scoring function and a neural network output designed for training RNNs to tackle sequence learning problems such as speech recognition. Instead of the need for per-frame labels, a CTC only needs “phoneme”-level labels whereby one phoneme can be mapped to multiple timeframes in the original speech audio. A CTC achieves this by transforming the network outputs into conditional probabilities and calculating the overall probabilities of all possible alignments that yield the same label sequence. It then finds the most probable sequence and corresponding alignment and uses that as the final output. The alignment can be represented by a set of segmented positions that separate each “phoneme” in a speech recognition problem. CTC’s ability to handle time sequence data and make predictions without the per-frame labels significantly broadened the applicability of RNNs.

This paper combines the LSTM and CTC to enable a novel WSOD by leveraging temporal classification. Next section introduces our methodology in detail.

Our feature extractor is built on a pretrained VGG16 neural network [Simonyan and Zisserman(2014)] that is trained on ImageNet [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei] using image-level labels. The feature map after the last convolutional block (convolution + ReLU) is fed into the proposed LSTM+CCTC for proposal generation and then the C-MIDN [Gao et al.(2019)Gao, Liu, Guo, Ye, Wan, You, and Fan] for classification.

Our PRN consists of four LSTMs that process the 1D feature map serialized by scan orders in four different directions (the four colored arrows on the feature map in Stage 2 of Figure 1). Each LSTM is connected to a CTC layer. The original CTC framework is designed for segmenting sequence data, while here we apply it to WSOD. To do so, we perform a transformation of the feature map with the size of $n\times n\times k$ to a 1D vector with a length of $n^{2}$. Each element is at $1\times k$ in dimension. This conversion makes the feature map suitable to serve as the input of the LSTM + CTC network. Different from the traditional CTC model which trains on and predicts a sequence of (different) labels, our model is based on a count-based CTC, or CCTC, the goal of which is to inspect objects without the need to differentiate object type. So this LSTM-CCTC can also be considered as a binary segmentation problem. The object type is identified by the classifier at the next stage of the WSOD pipeline. After critical points are located, initial proposals with different aspect ratios (1:1, 2:1, 1:2, 1:3 and 3:1) are generated. Then, proposals are gradually enlarged while keeping at the same shape (Figure 2) until they hit the border of the feature map.

In this stage, a region-based CNN is trained to classify proposals into different classes. In this work, we adopt C-MIDN [Gao et al.(2019)Gao, Liu, Guo, Ye, Wan, You, and Fan] as our WSOD network (Figure 1). This network couples two WSDDN [Bilen and Vedaldi(2016)] and contains iterative refinement in its instance classifier [Tang et al.(2017)Tang, Wang, Bai, and Liu]. The original WSDDN map proposal scores in a given image to an image-level classification confidence. Therefore, it can be trained solely under image-level supervision by optimizing a multi-class cross-entropy loss. However, the WSDDN often localizes the most discriminative object parts instead of the entire object because of the non-convex optimization. To address this issue, C-MIDN leverages a pair of WSDDNs, and the top-scoring proposals of the first WSDDN are removed from the input of the second WSDDN, preventing the second WSDDN from localizing the same proposal again. Furthermore, a segmentation map of the image is used to improve the robustness of the proposal removal process (Segmentation Guided Proposal Removal) (SGPR) to ensure that the full-context proposals will be kept. If the coverage rate between the segmentation map and the proposal is too small, it is more likely that there exist other tight proposals. Finally, proposals from the two WSDDNs are coupled to generate the final detection results. Next, the OICR training is adopted. Each stage is trained under the supervision of instance labels obtained from the previous stage. To obtain instance labels for supervision, given an image with a class label $c$, the proposal $j$ with the highest score for class $c$ will be used as the pseudo ground-truth BBOX. Besides labeling $j$, other proposals which have a high spatial overlap with $j$ will also be labeled as class $c$. The refinement strategy will give a preference to select the BBOX that contains the entire object instead of a part of it. It is very convenient to generate the pseudo ground-truth BBOX from our proposed RPN because these BBOXes (shown in Figure 2) share the same center, and many of them have a large overlap.

In the feature extractor implementation, we replaced the last pooling layer with a SPP layer and combined it with the proposed RPN and a WSOD network ([Gao et al.(2019)Gao, Liu, Guo, Ye, Wan, You, and Fan]). In the RPN, the LSTMs are unidirectional with 2 layers, with the size at 512 for the input layer, and 256 for the hidden layer. The outputs are connected to the same fully-connected layer with the output size of 2 for binary object detection. We followed the original settings in [Gao et al.(2019)Gao, Liu, Guo, Ye, Wan, You, and Fan] for training and testing the WSOD network. During training, since the proposals from the first several iterations are noisy, we trained the proposed LSTM-CCTC for 20 epochs and then connected it with the WSOD network for an end-to-end training. The number of objects in each image is transferred into a sequence (e.g. 3 is transferred into ’ooo’, where ’o’ refers to a target object of any class) to train the LSTM-CCTC. All the initialization for newly added layers used Gaussian distributions with 0-mean and standard deviation 0.01. For the optimization, we used Stochastic Gradient Descent (SGD) with a min-batch size of 2. The learning rate is 0.001 for the first 200 epochs and 0.0001 for the following epochs. The momentum and weight decay are set to 0.9 and 0.0005 respectively.

We also trained Fast R-CNN [Girshick(2015)] with top-scoring proposals generated by our proposed approach as the pseudo ground-truth. This is a common practice to improve the detection performance [Tang et al.(2017)Tang, Wang, Bai, and Liu, Diba et al.(2017)Diba, Sharma, Pazandeh, Pirsiavash, and Van Gool, Tang et al.(2018b)Tang, Wang, Wang, Yan, Liu, Huang, and Yuille, Gao et al.(2019)Gao, Liu, Guo, Ye, Wan, You, and Fan].

Datasets We evaluated our method on PASCAL VOC 2007 and VOC 2012 datasets [Everingham et al.(2010)Everingham, Van Gool, Williams, Winn, and Zisserman] which are two widely used benchmarks in WSOD. For both datasets, we combined training and validation images as the trainval set for training and used test images for testing. We generated the object count for each image from the BBOX annotations to train our proposed RPN and used image-level labels to train the Stage 3 WSOD network.

Evaluation metrics We used two performance metrics for evaluation: mean average precision (mAP) [Everingham et al.(2010)Everingham, Van Gool, Williams, Winn, and Zisserman] and correct location (CorLoc) [Deselaers et al.(2012)Deselaers, Alexe, and Ferrari]. mAP is a standard metric to evaluate the prediction accuracy and CorLoc measures the localization accuracy of a trained model.

Benchmarks We compare our method with seven other popular WSOD networks, including (1) WSDDN [Bilen and Vedaldi(2016)], one of the phenomenal CNN-based networks for WSOD; (2) OICR [Tang et al.(2017)Tang, Wang, Bai, and Liu], an improved WSDDN with a classifier refinement algorithm OICR; (3) WSRPN [Tang et al.(2018b)Tang, Wang, Wang, Yan, Liu, Huang, and Yuille], a CNN network focusing on generating high-quality proposals. This work shares the same goal as our proposed method; (4) C-WSL [Gao et al.(2018)Gao, Li, Yu, Morariu, and Davis], which uses per-class count as the weak labels for WSOD; and three most recent state-of-the-art WSOD solutions: (5) C-MIDN [Gao et al.(2019)Gao, Liu, Guo, Ye, Wan, You, and Fan], (6) C-MIL [Wan et al.(2019)Wan, Liu, Ke, Ji, Jiao, and Ye], and (7) MIL-OICR(PCL)+GAM+REG [Yang et al.(2019)Yang, Li, and Dou].

Table 1 shows the AP and mAP evaluated on PASCAL VOC 2007 dataset. It can be seen that our method (LSTM-CCTC+C-MIDN) outperforms all other related works. In particular, our proposed model achieves better performance (7.8% higher in mAP) than WSRPN [Tang et al.(2018b)Tang, Wang, Wang, Yan, Liu, Huang, and Yuille], which also aims at improving the quality of generated proposals. Our proposed method also beats the count-guided C-WSL (C-WSL+ODR) [Gao et al.(2018)Gao, Li, Yu, Morariu, and Davis] by 7.5% even with the use of weaker labels (total object count) than the per-class count used in C-WSL. We also integrated our proposed RPN into two well-performed WSOD frameworks: OICR[Tang et al.(2017)Tang, Wang, Bai, and Liu] and C-MIDN[Gao et al.(2019)Gao, Liu, Guo, Ye, Wan, You, and Fan]. The integrated networks yield a 6.3% and a 0.6% increase in mAP than [Tang et al.(2017)Tang, Wang, Bai, and Liu] and [Gao et al.(2019)Gao, Liu, Guo, Ye, Wan, You, and Fan], respectively. This owes solely to the introduction of LSTM-CCTC for proposal generation.

Our model also achieves the state-of-the-art performance by training Fast R-CNN with pseudo ground-truths. The statistics on CorLoc are shown in Table 2. The results show that our proposed WSOD achieves the best CorLoc among all methods compared, including those utilizing ensemble learning, such as WSRPN-Ens.+FRCNN[Tang et al.(2018b)Tang, Wang, Wang, Yan, Liu, Huang, and Yuille].

The same experiments were conducted on VOC 2012 and results are shown in Table 3. The results verify that our proposed model achieves better performance than the other popular WSOD models.

We also conducted ablation experiments on PASCAL VOC 2007 to analyze the impact of different factors on the performance of our proposed network.

Ways of feature map serialization The first factor is ways of feature-map serialization, or more specifically, the number of scan orders used in serializing feature map at Stage 2. As described in Section 3.2, we made a point that applying LSTM to serialized data derived from multiple scanning directions will lead to identification of more (and accurate) critical points falling on or near a target object. This is based on the assumption that the “temporal” and contextual patterns exerted by different objects may be more predominantly shown in data serialized by different scan orders instead of a fixed one. We conducted experiments to apply only one way of serialization (direction in red in Figure 1), and two ways of serialization (directions in both red and black in Figure 1) and all four ways of serialization (Figure 1). The mAP and CorLoc we obtained for the three scenarios are 38.1% mAP and 52.4% CorLoc for scenario 1, 44.6% mAP and 63.1% CorLoc for scenario 2 and 53.1% mAP and 70.0% CorLoc for scenario 3. This result verifies our assumption.

Different proposal generation methods Previous studies [Hosang et al.(2015)Hosang, Benenson, Dollár, and Schiele] argued that the quality of proposals plays an important role in affecting the OD performance. This statement is especially true in a WSOD context when the exact BBOX labels are not available. Here, we compare our region generation method with typical methods such as SS and EB. To make a fair comparison, we integrated these comparing approaches into our learning framework (LSTM-CCTC+C-MIDN), with the replacement of Stage 2 by SS and EB, respectively. To ensure other networks to achieve their best possible performance within a reasonable training time, 2k proposals were generated for each method. Only an average of 200 proposals were generated by our proposed method instead. The results are: 52.6% mAP and 68.7% CorLoc for SS and 49.5% mAP and 66.4% CorLoc by EB. Our method (53.1% mAP and 70.0% CorLoc) clearly outperforms commonly used proposal generation techniques.

Proposal recall Figure 3 shows the proposal recall with ground truth bounding boxes at different IoU levels [Tang et al.(2018b)Tang, Wang, Wang, Yan, Liu, Huang, and Yuille]. According to [Hosang et al.(2015)Hosang, Benenson, Dollár, and Schiele, Chavali et al.(2016)Chavali, Agrawal, Mahendru, and Batra], a high recall is not a necessary condition of high detection mAP. We simply use this result to measure the quantity and quality of the generated proposals by our proposed network. In figure 3, we observe that our recall is not as high as other methods when the IoU level is low. It is because our algorithm highly relies on finding objects in one-time scanning instead of an exhaustive search. Compared to an exhaustive search, our network might not be able to locate as many objects as they do, however, once an object is located, our network generates a proposal of much better quality. This can be seen in the figure that our curve decays slower than other methods when IoU level gets higher. Our algorithm proposes multiple candidate boxes with different sizes and ratios around an object location such that it is more likely to have a better-quality proposal. The recall curve of RPN [Ren et al.(2015)Ren, He, Girshick, and Sun] is used as a reference here showing the difference between strong supervision (with bounding box information) and weak supervision.

This result proves that our network can correctly label the position of objects and once we have correct aspect ratios of objects, the network generates tight proposals pretty well. One concern of increasing the number of proposals is that it also increases the computation effort. However, our total number of proposals is still far less than traditional methods like SS [Uijlings et al.(2013)Uijlings, Van De Sande, Gevers, and Smeulders] and EB [Zitnick and Dollár(2014)]. Only an average of 200 proposals per image were generated by our proposed method. In addition, unlike other works which generate a fixed number of proposals for each image, our total number of proposals for each image is proportional to the number of interested objects in the image. Our network achieves a good trade-off between computation effort and detection accuracy.