Hallucination Improves Few-Shot Object Detection

Two-stage Fine-tuning Approach (TFA): A two-stage fine-tuning approach is introduced [38] under the widely-used Faster R-CNN framework, which significantly improves few-shot detection performance. As a serial detector, Faster R-CNN consists of a backbone, a region proposal network (RPN), a region of interest (RoI) pooling layer, an RoI feature extractor, a bounding box classifier, and a bounding box regressor (Figure 2). The last four components together construct the RoI head. The backbone extracts image features, which are passed through the RPN to produce promising areas with potential objects. The RoI head then transforms, classifies, and refines potential object boxes into labeled boxes. TFA modifies the standard Faster R-CNN by using a cosine similarity based classifier to reduce intra-class variance for few-shot learning. TFA uses an ImageNet [29] pre-trained ResNet-101 with a feature pyramid network [21] as the backbone. As shown in Figure 2, in the first stage, TFA trains the whole model using base class images. In the second stage, only the classifier and the bounding box regressor in the RoI head are fine-tuned using novel class instances, with the rest of the model frozen.

Cooperating RPN’s (CoRPNs): An improved proposal generation mechanism based on RPN ensemble is proposed in CoRPNs [48], which produces highly informative proposal boxes for few-shot detection. As shown in Figure 3, CoRPNs train multiple distinct but cooperating RPN’s so that if one RPN misses a highly informative proposal box, another one will get it. Except the proposal generation procedure, CoRPNs share the same model architecture and training strategy with TFA. Specifically, CoRPNs train $N$ RPN’s by a modified RPN classification loss

$$\mathcal{L}=\mathcal{L}^{j^{*}}_{\text{CE}}+\mathcal{L}_{\text{div}}+\mathcal{L}_{\text{coop}},$$

(1)

where $\mathcal{L}^{j^{*}}_{\text{CE}}$ is a binary cross-entropy loss of a selected RPN $j^{*}$. CoRPNs select the most certain RPN for every box. For instance, each anchor box will get the RPN $j^{*}$’s score that has a probability closer to either 0 or 1. The selected RPN $j^{*}$ gets this box’s gradient at training.

The divergence loss $\mathcal{L}_{\text{div}}$ encourages RPN’s to be different, and the cooperation loss $\mathcal{L}_{\text{coop}}$ enforces RPN’s to cooperate by setting a lower bound for every RPN’s response to a foreground box. $\mathcal{L}_{\text{div}}$ is defined as

$$\mathcal{L}_{\text{div}}=-\log(\det(\Sigma(\mathcal{F}))),$$

(2)

which is the negative log of the covariance matrix on every RPN’s prediction to proposal boxes. For foreground box $i$ and RPN $j$, the cooperation loss is

$$\mathcal{L}_{\text{coop}}^{i,j}=\max\{0,\phi-f^{j}_{i}\},$$

(3)

where $\phi$ is the lower bound of every RPN’s response to a foreground box, and $f^{j}_{i}$ is RPN j’s response to the box i. $\mathcal{L}_{\text{coop}}$ is averaged over all RPN’s and foreground boxes.

We introduce a hallucinator network $H$ with parameters $\phi$ that learns to generate additional examples for novel classes by leveraging the shared within-class feature variation from base classes.

As shown in Figure 4, hallucination happens in the RoI head feature space. The hallucinator takes as input available training examples and generates hallucinated examples. The set of hallucinated examples $S_{\text{gen}}$ is then treated as additional training examples for learning the classifier on novel classes. Specially, given a seed example $x_{i}^{c_{k}}$ of category $c_{k}$, the hallucinator generates a hallucinated example by

$$\widetilde{x^{c_{k}}_{i}}=H(\mu_{c_{k}},x_{i}^{c_{k}},\varepsilon;\phi),$$

(4)

where $\mu_{c_{k}}$ is the class prototype of $c_{k}$, which is the mean of all instances from $c_{k}$, and $\varepsilon$ is a per-example noise vector. Note that our hallucinator formulation is different from [40]: we introduce an additional input $\mu_{c_{k}}$ to explicitly capture the global category information. For $\widetilde{x^{c_{k}}_{i}}$, its label $y_{i}$ is the same as the seed example’s category $c_{k}$. Assuming that we already have a trained box classifier, we directly use the classifier’s (cross-entropy) classification loss on all hallucinated examples to train the hallucinator:

$$L_{\text{halluc}}(\phi)=\sum_{i\in S_{\text{gen}}}\left(-w^{T}_{y_{i}}\widetilde{x^{c_{k}}_{i}}+\log\sum_{k}e^{w^{T}_{k}\widetilde{x^{c_{k}}_{i}}}\right),$$

(5)

where $w_{y_{i}}$ and $w_{k}$ are the already-learned classifier’s weights on corresponding categories. This hallucination loss enforces the hallucinated examples to loosely “agree with” the trained classifier. In addition, since we take as input class prototypes, our hallucinator is implicitly regularized and thus not simply copying the seed examples. Compared with [40] which uses a complicated meta-learning process, we substantially simplify the training procedure.

As illustrated in Figure 5, we propose an EM-style (expectation–maximization) training procedure to train the hallucinator and the detector’s classifier: when training the hallucinator, the classifier is frozen; when training the classifier (and the box regressor), the hallucinator is frozen. Compared with end-to-end joint training, this iterative strategy improves the cooperation between the hallucinator and the classifier, thus producing examples that are more useful for building the few-shot classifier.

Training on Base Classes: In the stage of training on base classes, we first train the plain detector without the hallucinator in a standard way as before. We then insert the hallucinator into the detector, and freeze all model components except the hallucinator. Now, the hallucination will be performed in the pre-trained, fixed RoI head feature space. And the classifier is already trained on base classes using all available examples. We then use this base class classifier to guide the training of the hallucinator.

Note that, different from few-shot classification, the proposal generation process in detection produces several proposal boxes around an object, making a few training examples available even in the 1-shot scenario. We randomly sample these training examples as seed examples for the hallucinator. Consider a training batch $S$ which consists of $n$ base classes. For each class in the batch, the hallucinator generates $m$ examples, and $m$ is fixed for all batches. The set of hallucinated examples $S_{\text{gen}}$ is used to train the hallucinator based on the loss function (5).

Fine-Tuning on Novel Classes: After training the hallucinator on base classes, we move on to the stage of fine-tuning the classifier and hallucinator on a few-shot dataset containing both base and novel classes. We freeze all model components prior to the hallucinator. Each training batch $S$ initially consists of an imbalanced set of foreground box examples $S_{\text{pos}}$ and background box examples $S_{\text{neg}}$, with background examples being the majority: $S=[S_{\text{pos}};S_{\text{neg}}]$. The hallucinator generates a set of additional examples $S_{\text{gen}}$ for novel classes. Without changing the number of examples in total, we randomly replace $|S_{\text{gen}}|$ background examples by $S_{\text{gen}}$ to obtain a refined training batch

$$\widetilde{S}=[S_{\text{pos}},S_{\text{gen}};S^{\prime}_{\text{neg}}].$$

(6)

By doing so, we also partially alleviate the imbalance issue between foreground and background examples. After training the classifier on the refined dataset with hallucinated examples, we fine-tune the hallucinator using the classifier based on Eq. (5), then we use the fine-tuned hallucinator to fine-tune the classifier again, and so on. We stop this procedure after one or two iterations (\ie, the classifier is fine-tuned at most twice); empirically, we found that additional iterations do not further improve the performance.

Datasets and Evaluation Metrics: We evaluate on two widely-used few-shot detection benchmarks: MS-COCO [22] and PASCAL VOC (07 + 12) [6]. For a fair comparison, the same base/novel category splits, train/test splits, and novel class instances [18, 38] are used for training and evaluation. Consistent with recent work [18, 38], we report AP50 for three different base/novel splits under shots 1, 2, 3, 5, and 10 on PASCAL VOC; we report AP, AP50, and AP75 under shots 1, 2, and 3 on COCO. Note that we particularly focus on the extremely-few-shot regime.

Baselines: We mainly focus on the comparison with two state-of-the-art few-shot detectors – TFA [38] and CoRPNs [48], and evaluate the effectiveness and generalizability of our hallucinator when combined with them. For both our models and the main baselines, we use Faster R-CNN [28] as our base model. Following [38], we use an ImageNet pre-trained ResNet-101 with a feature pyramid network [21] as the backbone. All our experiments, including those with the main baselines, have ground-truth boxes appended as training examples in the RoI head. As reported in [38], including ground-truth boxes leads to a 0.5% AP gain on COCO. In addition, we compare with a variety of baselines, including concurrent work on few-shot detection.

Evaluation Procedure: Both our models and the main baselines are $(|C_{\text{b}}|+|C_{\text{n}}|)$-way few-shot detectors. And we evaluate them on both base and novel classes under the standard evaluation procedure [38], as described in Section 3. Some other baselines like [17] were initially evaluated under different procedures. For a fair comparison, all reported numbers for those methods are the re-evaluated results under the standard evaluation procedure.

Note that, following [38], the fine-tuning stages on PASCAL VOC and COCO are slightly different. On PASCAL VOC, the classifier’s weights on novel classes are randomly initialized and directly trained using a balanced dataset with base and novel classes. On COCO, we first train a $|C_{\text{n}}|$-way classifier on novel class instances. The trained classifier is used as an initialization of the classifier’s weights on novel classes. We then train a $(|C_{\text{b}}|+|C_{\text{n}}|)$-way classifier using a balanced few-shot dataset with base and novel classes.

Hallucinator Architecture: We use a two-layer MLP (multi-layer perceptron) with ReLU as the hallucinator. Given that the inputs to the hallucinator are a class prototype, a seed example, and random noise, the input size is three times the feature size; the output size of each linear layer is the same size as the input feature. The seed example is randomly sampled from all examples in a training batch. The input noise is dataset-dependent. For each dataset, we calculate the mean and standard deviation of pre-trained input features and use those to sample the input noise.

We pre-compute base class prototypes using all available examples before training the hallucinator. During training the hallucinator, we do not update these base class prototypes. In a different manner, we construct novel class prototypes dynamically, when using the learned hallucinator to train classifiers on novel classes. Specifically, novel class prototypes are computed by using both the training and hallucinated examples (in contrast to that only training examples are used for base class prototypes). For a novel class, whenever a new training example comes in or a new example is generated, its corresponding prototype will be updated. Despite somewhat inconsistency between base and novel classes for constructing prototypes, we found that this strategy exploits all available examples and empirically achieves the best performance.

Hallucinator Initialization: Following a similar strategy as in [40], we initialize the hallucinator by using block diagonal identity matrices plus small random noise. The initialization noise is sampled from a normal distribution with a zero mean and a standard deviation of 0.02. As a form of regularization, this identity initialization ensures that the initially hallucinated examples are not too far away from the seed examples and thus do not degrade the performance.

Hyper-Parameter Settings: On PASCAL VOC, we train the hallucinator with batch size 16 and learning rate 0.02 for 8,000 iterations. We decay the learning rate by ratio 0.1 at 2,000 and 6,000 iterations. On COCO, we train the hallucinator with batch size 64 and learning rate 0.02 for 21,200 iterations. We decay the learning rate by ratio 0.1 at 6,400 and 19,200 iterations.

Number of Hallucinated Examples: When training the classifier on novel classes, the hallucinator produces a fixed number ($m$) of examples per category in each training batch. For simplicity, we set $m$ to be the same as the average number of per-class training examples (\ie, RoI features), averaged over the training batches. On PASCAL VOC, there are roughly 20 training examples per class (with batch size 16), and we thus hallucinate 20 examples per class accordingly for all experiments. We keep this number on COCO as well. In the ablation study, we show that by tuning the number of hallucinated examples, the performance gets further improved. Note that this number should change when the number of images per batch varies.

Comparison with Main Baselines TFA and CoRPNs: Tables 1 and 2 summarize the results for novel classes on PASCAL VOC and COCO, respectively. We have the following key observations.

(1) Hallucination improves the performance over the main baselines by large margins in the extremely-few-shot regime. On PASCAL VOC, hallucination yields substantial improvements in 1-shot and 2-shot scenarios. On the more challenging COCO benchmark, hallucination consistently improves performance when combined with both baselines.

(2) Our hallucination strategy is general and applicable to different types of detectors regardless of RPN outputs. This is because hallucination provides additional, useful sample variation that is independent of RPN outputs.

(3) On PASCAL VOC (Table 1), hallucination demonstrates slightly different behaviors when combined with CoRPNs or TFA: while hallucination consistently improves CoRPNs in the extremely-few-shot regime, it is not always helpful for TFA in some cases (though hallucination also does not hurt). A possible explanation is that our hallucinator tends to be conservative, making the hallucinated examples still close to the original seed training examples. Hence, in the cases (\eg, novel sets 2 and 3) where the training and test examples are quite distinct, the hallucinated examples produced from the training examples could not help build a better classifier for detecting the test examples. In such cases, a more aggressive hallucinator is desired to generate examples that are further away from the seed examples. The ablation study in Section 4 investigates this issue in more depth and supports our observation.

(4) The performance gain of hallucination diminishes as the number of training examples increases. In higher-shot scenarios, our current hallucination does not bring in additional within-class variation beyond the given training examples, and is thus no longer beneficial. Note that, however, hallucination does not hurt the performance in 5/10-shot scenarios – performance variation is within 95% confidence interval over multiple random samples according to [38]. Given that a single hallucinator is trained in our case, we might need different hallucinators or hallucination strategies in different sample size regimes.

Comparison with Other State-of-the-Art Methods: Tables 1 and 2 show that our approach is superior or comparable to other few-shot detection methods. In particular, we achieve very competitive results in the extremely-few-shot regime. We believe that our effort is orthogonal to concurrent work; our hallucination strategy can be combined with other methods to further improve their performance.

Base Class Performance: Table 3 presents the detection results on base classes, evaluated after fine-tuning with novel class instances. We observe that hallucination improves novel class performance at the expense of slight degradation of performance on base classes. This issue could be potentially addressed by using a smaller fine-tuning learning rate for the classifier components of base classes (which have been already well-trained). More notably, compared with other state-of-the-art methods such as MPSR [43] and FsDetView [44], we achieve a significantly better trade-off between novel and base class performance.

EM vs. Joint Training Procedures: Table 4 shows that, with the same hallucinator architecture, our EM-style training procedure significantly outperforms joint training, indicating that joint training might be greedy in this case. In addition, 2 EM-iterations outperform 1 iteration, and we found that more iterations do not improve the performance.

Conservative vs. Aggressive Hallucinators: In the main experiments, we show that a conservative hallucinator is not always helpful and in some cases, we need a more aggressive hallucinator. Here we build such an aggressive hallucinator by changing the hallucination space. As shown in Figure 4, the original hallucinator generates examples directly in the operational space of the classifier (\ie, in the feature space after the box head and right before the classifier); this makes it conservative and reluctant to produce the kinds of examples that drastically change the classifier decision boundaries. By contrast, our new hallucinator generates examples in the feature space before the box head and accordingly, it becomes a three-layer convolutional network. Now, the box head makes allowances for errors in the hallucination, and the hallucinator becomes more aggressive and allows radical change of classifiers. We also make some additional modifications as follows: (1) we jointly train the aggressive hallucinator with other model components, and (2) we use a cosine prototypical network loss [34] computed on held-out validation examples with hallucinated prototypes as additional guidance to train the hallucinator. Table 5 shows that the aggressive hallucinator significantly helps the cases where the performance of the conservative hallucinator is lagging. Importantly, neither of the two hallucinator variants hurts the performance.

Number of Hallucinated Examples: Our hallucinator generates a fixed number of examples per class in each training batch (20 examples in the main results). Table 6 investigates the impact of the number of hallucinated examples. As it increases from 0 to 20, the performance gradually improves and then saturates and drops slightly. Note that the drop is still within 95% confidence interval over multiple random samples according to [38]. This is because even if we produce a small number of hallucinated examples per training batch (\eg, 3), after many iterations, the cumulative number of hallucinated examples is large enough.

Results with Fully-Connected Classifiers: We use cosine classifiers in the main results. Table 7 shows detection results using fully-connected classifiers. Hallucination improves CoRPNs in all 1-shot cases, suggesting the effectiveness of hallucination irrespective of the classifier choice.

Performance Gain Analysis: We investigate where our performance gain comes from. The detection AP can be improved in two ways: (1) more boxes are correctly identified, \ie, true positive rate goes up; (2) fewer boxes are misclassified, \ie, false positive rate goes down. As shown in Table 8, hallucination both increases the number of true positives in most cases and always decreases the number of false positives. We find that the scale of improvements on false positives is larger than the scale of improvements on true positives. A possible explanation is that hallucination generates close-to-boundary examples, and improves decision boundaries. Many misclassified boxes are eliminated, since we have a better estimation of decision boundaries.