Semi-Weakly Supervised Object Detection by Sampling
Pseudo Ground-Truth Boxes

Models for object detection can be broadly classified into two-stage[21, 16] and one-stage [20, 17] object detectors. In both cases, supervised object detection requires class and bounding box labels for each instance present in the image[11]. Two-stage object detectors have a first stage that extracts RoIs (candidate object regions) whose reliability as a potential candidate region is quantified by their objectness score. Earlier approaches extract these RoIs using low-level image features in R-CNN[11], Fast R-CNN[10], etc. Later, end-to-end two-stage models emerged as more accurate detectors, where an additional learnable head called RPN (Region Proposal Network) is used to regress candidate regions[21, 16, 5]. In contrast, one-stage object detectors avoid the RoI extraction stage, and classify and regress directly from the anchor boxes. They are generally fast and applicable to real-time object detection[20, 19, 17]. Though the two-stage detectors are typically slower compared to their one-stage counterparts, extracting reliable candidate region in the first stage provides an edge in terms of localization accuracy. Recently, anchor-free detectors[30, 7] have grown in popularity since they dissociate from hand-designed anchor box selection and matching process, and provide even faster detectors.

Semi-supervised object detectors rely on a small subset of labeled images and a large collection of unlabeled images to train a detector[29]. They have been explored in many different forms. Recently, detectors trained in a student-teacher fashion have gained popularity in the research community[32, 27]. In this setting, a teacher model is trained first using the available annotated data, and then used to produce pseudo-labels for unlabeled data. Once the pseudo-labels are obtained, a student model is trained using the combined dataset. Though our proposed method also make use of the concepts of pseudo-labeling, our approach differs from these methods because we don’t have two separate networks, and multiple training stages, making it a simpler solution for semi-supervised detection.

Weakly-supervised detectors, on the other hand, only rely on image-class labels[28, 1, 6]. Research in weakly-supervised methods have progressed greatly in recent years[22]. However, a fundamental problem is their inability to distinguish the entire object from its parts and context. As a result, the detectors often produce bounding boxes on discriminative object parts, failing to distinguish the object boundaries when multiples objects of the same class are spatially adjacent. They produce imprecise localization that do not differentiate object boundaries from its context. Our proposed method also learns from a vast majority of weakly-labeled images, but, combined with a few fully-annotated images, providing better objectness distillation from the fully- to weakly-annotated images. We argue that this is the most practical setting, as we can typically annotate a small number of images with bounding boxes, and supply the remaining with weak annotations, or no annotations at all. Some approaches also attempt to leverage the additional, easy to obtain weak supervision for the unlabeled data. In particular, they used weak image-level labels[9] or point annotations[4] for the unlabeled images. Though this required additional annotation effort, they are easy to obtain, and may provide substantial improvements. Our approach also relies on weak image-class labels.

Self-training has emerged as a popular approach in weakly and semi-supervised object detection[28, 22, 4]. In these approaches, a detector is first trained on the available data and then the obtained detections from the first training are used for training a refined model. This procedure can be repeated often multiple times. However, these methods perform self-training with pseudo labels obtained offline, e.g., a Faster R-CNN network training with detection result of weakly-supervised detectors as pseudo labels[28, 22]. In contrast, our method performs self-training using the pseudo labels for weakly labeled images, but in an online fashion. In particular, for each iteration, our approach samples pseudo GT boxes for each given weak label using the accumulated semantics from a score propagation block. Given this online nature of the self-training, the training is a simple one-stage process.

For the images that are strongly annotated (.i.e bounding box annotation for each object present), the learning step is straightforward. Given an input image $I$, the ground truth (GT) annotations are defined with the bounding box positions $\mathcal{B}=\{b_{0},b_{1},\cdots\ b_{N}\}$ and corresponding classes $\mathcal{C}=\{c_{0},c_{1},\cdots\ c_{N}\}$. $b=(x_{0},y_{0},x_{1},y_{1})$ is a vector with 4 values that represents for instance the top left and bottom right corner of a box, while $c\in\mathcal{C}$ is a discrete value that represents the object category. In our experiments we use faster RCNN as detector [21], and thus use a loss as:

For each GT bounding box $b_{i}$ it generates a loss based on the scores $f_{c_{i}}$ and overlap of the obtained detections $d_{j}$. $L_{cls}$ and $L_{reg}$ denote the classification and localization loss, respectively. $N_{cls}$ and $N_{reg}$ are the normalization factors which depends on the number of foreground and background RoIs considered. $\lambda$ is a hyperparameter that controls the relative importance of the classification and localization loss. Note that the exact form of loss can vary according to the fully supervised detector architecture used in the model, but our approach is independent of the specific fully supervised loss.

In case of weak supervision, we know the object classes that are present in the image $c$, but not the bounding box locations $b_{i}$. Thus, the general approach of weakly supervised models is to learn during training what are the regions of the image that are more likely to contain the object of interest. A typical approach for doing that is to compute the classification loss for a given class $c$ and an image $I$ as a weighted sum of scores:

where $f_{c}(p_{l},I)$ is the score of the detector for ground truth class $c$ and object proposal $p_{l}\in\mathcal{P}$ of to the image $I$, and $w_{l}$ is a normalized weight associated with each object proposal $p_{l}$. The weights associated with each proposal $w_{l}$ for a image $I$ are computed as normalized scores:

with $T$ being the temperature parameter that defines the sharpness of the weight distribution and is a hyper-parameter of the learning approach. In this way, boxes with higher score will have more impact on the learning and the learning will focus more and more on the locations of the image that are more likely to contain the object of interest. While this formulation works well, it is computationally expensive because it has to evaluate at each training iteration and for each image all box locations $p_{l}$.

Here, we propose to approximate the weighted sum of scores $h$ with Monte Carlo sampling, in which instead of computing the sum over entire bounding box locations $l$, we uniformly sample $K$ boxes:

where $\theta_{k}$ is the weight associated with the bounding box $k$. This allows us to compute only $K$ evaluations of the expensive $f$, while using an unbiased estimation of the weakly supervised scoring function. We call these samples pseudo GT bounding boxes because as we will see in the next section, these samples can be pass to the detection algorithm as ground truth annotations. This allows our algorithm to use any off-the-shelf object detector without modification.

However, when sampling, the weights $\theta_{k}$ associated with each bounding box score cannot be computed directly because it is the normalized version of the object score $f_{c}(p_{k},I)$ for the box $p_{k}$, but we do not have all scores for the normalization factor, as we sampled only a few boxes. Instead, for each image $I$ we keep in memory the scores $s_{l}$ associated with the bounding box $p_{l}$ and update only the score of the $K$ sampled bounding boxes. Then, $\theta_{k}$ will be computed as the normalized version of $s_{k}$:

As $\theta_{k}$ approximates $w_{l}$, we see that when the scores $f$ do not change anymore (i.e. at convergence), $\hat{h}$ becomes $h$. Therefore $\hat{h}$ is an unbiased estimation of $h$. While this approach would work, sampling uniformly any possible image bounding box proposal $p_{l}$ would make the learning very slow because most of the time the sample would not come from the object of interest. Instead, in this work we use an importance sampling approach. We use $\theta_{k}$ as sampling probabilities associated with a multinomial distribution $\mathcal{M}(\theta_{k})$ to sample bounding boxes, so that the bounding box proposal $p_{k}$ would have a probability $\theta_{k}$ to be sampled. In this case, in order to maintain the same estimation of $h$ we need to divide by the sampling probability $\theta_{k}$. Thus the final estimation of $h$ will be:

which is again an unbiased estimator of the classification score of an image, but with lower variance. Thus, the final weakly supervised loss $L_{W}$ for an image $I$ is the same as in Eqn.(1), but with the ground truth bounding boxes $b_{i}^{*}$ changed to the pseudo GT boxes sampled from the object proposals $\mathcal{P}$.

While the fully supervised object detector uses ground truth boxes that are correct, the weakly supervised counterpart estimates the object box location during training and therefore the estimation can be noisy. Thus, when learning with strong and weak labels we might want to set a hyper-parameter value that balances the relative importance of the two losses. In this case the final loss is $L=L_{F}+\lambda L_{W}$. As the aim of this approach is to avoid to directly modify the loss on the detector, in this case we express the weight $\lambda$ with a sampling ratio. Thus, we use a ratio parameter $r$ that controls the amount of training data from the fully and weakly labeled pool of data. For instance $r=0.6$ means that we sample with probability $0.6$ from the pool of the fully-labeled samples and $0.4$ from the weakly-labeled samples. This approach has the advantage of not changing the internals of the underlying detector such as loss function or additional regularization etc. With this design, we can feed both the fully annotated and weakly annotated images in parallel to the model and train it in a single stage.

In this section we present in more detail the different parts of the proposed learning algorithm as summarized in Alg. 1. For the sake of simplicity the algorithm is shown for the case of a single image $I$, but it could be trivially extended to a batch of images.

For supervised samples our algorithm uses directly the bounding box annotations $\mathcal{B}$ and the corresponding classes $\mathcal{C}$ for inference (detect). For weak supervision, the inference is performed on pseudo GT annotations ($\hat{\mathcal{B}},\hat{\mathcal{C}}$) that are obtained by sampling object proposals (sample). Then, the obtained detections $\mathcal{D}$ and scores $\mathcal{S}^{D}$ are used to update the proposal scores ($\mathcal{S}^{P}$). In both cases, the obtained detections $\mathcal{D}$ and scores $\mathcal{S}^{D}$ are used to compute the loss $L$ and update the recognition model (backprop).

In our basic model we use around 2000 object proposals obtained from the selective search algorithm [31] in order to have a high detection recall. However, keeping so many proposals, means that we need to keep a large set of scores. This will make the algorithm slow and more noisy in the beginning of the training as there are many possible regions to explore. In the experiments, we tested to use a Class Activation Map (CAM) model [25] to reduce the initial number of proposals. In practice, for each image and for each class present in an image, we extract its CAM. Then, for each CAM region, only the proposals that overlap at least $\rho$ with that CAM region are kept. The final set of proposals will be the union of the proposals selected for each class.

We extend the weakly semi-supervised approach proposed to a normal semi-supervised approach, in which part of the data is fully supervised and the rest has not supervision at all. For doing that, we keep the same structure of the previous approach, but for the non-annotated images, instead of using ground truth labels, we use the labels provided by the image classifier already used to compute the CAMs. As we will see in the experiments, this will provide almost the same level of performance as the semi-weakly supervised but without the need of weak labels.

During training, the shorter edge of the input images are randomly rescaled within {480,576,688,864,1200}. We only use horizontal flipping for data augmentation. Object proposals are extracted using the selective search algorithm [31]. Typically, from an image, up to 2000 object proposals are extracted for good recall of all the object instances. Images are normalized with $\textrm{mean}=[0.485,0.456,0.406]$ and $\textrm{std}=[0.229,0.224,0.225]$, as in ImageNet training [24]. The network is trained on NVIDIA V100 GPU with 32GB memory.

Several ablation studies are conducted in order to asses the individual components of our proposed model. First, we study the impact on performance of different ways of defining the sampler and score propagation module. Then, we analyze the impact of learning with a growing amount of annotations. Finally, we study the contribution of different types of errors made by the model. All of the these studies are conducted on PASCAL VOC 2007 by training the model using its trainval set, and testing on its test set. We used 10% bounding annotations in this analysis, while the rest of the images are weakly annotated.

Table 4 shows a comparison of our method with state-of-the-art methods for semi- and weakly-supervised learning of object detectors. Pascal VOC 2007 is used as the fully-labeled data, while VOC 2012 is used as the unlabeled or weakly-labeled set. We first evaluate our method for semi-weakly supervised training. The only other method performing Semi-weakly supervised learning is WSSOD [9]. In this setting our method outperforms it, while being also more flexible, as it can be used with an detector model. Compared to the model trained only on VOC07 (lower bound), we observed a significant improvement (5.0%) when using additional weak labeled data, approaching a model with full annotations in both datasets (upper bound).

We then compare our model to the state-of-the-art in plain semi-supervised settings. To report the results of our method in the semi-supervised settings, we train a classifier on the available fully-labeled images, and used that classifier to obtain weak image-level labels for the unlabeled images. This requires training of an additional classifier, but it is less expensive than the detector pre-training used in most of the semi-supervised methods. Results in the table indicate a significant improvement in terms of performance, with the additional moderate cost of collecting weak image-level labels. In the semi-supervised case also, our method shows an improvement of 3.4%, outperforming most of the methods in terms of mAP and AP 50.