Entropy-based Active Learning for Object Detection
with Progressive Diversity Constraint

As stated, the majority of the studies on active learning in computer vision target on image classification and are mainly categorized into diversity-based [27, 1] and uncertainty-based [10, 38] ones.

The diversity-based methods screen a subset of samples to represent the global distribution by clustering [24] or matrix partition [12] techniques. Core-set [27] defines active learning as a core-set selection problem and adopts $k$-center approximation. To improve efficiency, CDAL [1] replaces distance based similarity with the KL divergence. Those methods are theoretically complete but computationally inefficient when dealing with high-dimensional data.

The uncertainty-based methods select ambiguous samples which are regarded as the most informative to the entire dataset [16, 10, 3, 38]. Many efforts are made to estimate the data uncertainty, e.g. the entropy of class posterior probabilities [16]. In this case, [10] introduces Bayesian CNNs as an expert; [3] employs deep ensembles and Monte-Carlo dropout; and Learn Loss [38] proposes a task-free image-level loss prediction module. The methods above are efficient, but bring in redundant samples for annotation.

Some alternatives [15, 7, 36] combine the advantages of both types. With the uncertainty and diversity scores, [7] simply chooses the minimal; [37] emphasizes the diversity at early cycles and moves to the uncertainty gradually; [14] views the fusion as a multi-armed bandit problem and re-weights different scores. VAAL [30] performs uncertainty estimation on whether a data point belongs to the labeled or unlabeled pool, and acquires the samples most similar to the latter. SRAAL [40] further exploits the uncertainty estimator and the supervised learner to enclose annotation information. BADGE [2] models the uncertainty and diversity by the gradient magnitudes and directions from the last layer respectively. The hybrid methods achieve promising results and suggest a new fashion for other tasks.

Object detection has been greatly progressed by CNNs [29, 13] in the past few years mainly under the one-stage [21, 19, 32] and two-stage [11, 25] frameworks. As detection annotation is more expensive and time-consuming, active learning comes into focus in this branch, and preliminary attempts demonstrate its necessity [4, 34, 23, 26, 17]. Meanwhile, with both object category and location to predict, this task is more challenging.

Both the diversity-based and uncertainty-based methods have been recently adapted to object detection, and they extend direct image-level decision by integrating instance-level predictions. For the former, CDAL [1] represents the image using the detection features after spatial pooling to approximate this process. In spite of certain potential, a substantial performance gain requires global instance-level feature comparison, which incurs a vast complexity. For the latter, Learn Loss [38] employs holistic image-level features for uncertainty estimation and with the task-free loss prediction module, it directly evaluates how much information an unlabeled image contributes. MIAL [39] selects the image by measuring its uncertainty based on that of the top-$K$ instances re-weighted in a multiple instance learning framework, with noisy ones suppressed and representative ones highlighted. They ignore instance-level correlation within the whole data pool and thus deliver much redundancy. To address the issues above, this paper presents a way to jointly use their strengths to advance object detection.

In this subsection, we describe the details about our proposed hybrid framework, specifically designed for active learning on object detection.

As shown in Fig. 1, the proposed framework mainly consists of three modules: uncertainty estimation using the basic detection entropy, Entropy-based Non-Maximum Suppression (ENMS) and the diverse prototype (DivProto) strategy. The basic detection entropy in Eq. (2) is adopted to quantitively measure the image-level uncertainty from object instances. ENMS is subsequently presented to remove redundant information according to the entropy, thus strengthening the instance-level diversity within images. DivProto further ensures the instance-level diversity across images by converting it into the inter-class and intra-class diversities formulated with class-specific prototypes.

To be specific, the hierarchy of our method for diversity enhancement is illustrated in Fig. 2. The overall instance-level diversity (a) is divided into the intra-image diversity (b) via ENMS and the inter-image diversity is accomplished by DivProto, which is then decomposed into the inter-class and intra-class ones as shown in (c) and (d), respectively. By virtue of this progressive way, the diversity constraints on the predicted instances are effectively undertaken. The details are elaborated in the rest part.

As Eq. (2) depicts, the basic detection entropy $\mathbb{H}(I_{i}|D_{\mathcal{S}})$ is a simple sum of the entropy of the candidate bounding boxes. Nevertheless, existing object detectors often generate a large amount of proposal bounding boxes with heavy overlaps, incurring severe spatial redundancy and high computational cost. This issue can be partially mitigated by applying Non-Maximum Suppression (NMS) [25, 21], based on which bounding boxes belonging to the same instance are merged to a unified one. But NMS cannot deal with the instance-level redundancy, i.e. instances with similar appearances presenting in the same context, which is supposed to be reduced in active acquisition.

To overcome this shortcoming of NMS, we propose a simple yet effective Entropy-based Non-Maximum Suppression (ENMS) as a successive step of NMS for instance-level redundancy removal. Specifically, we first compute the following Cosine distance $\rm{Sim}(\cdot,\cdot)$ to measure pair-wise inter-instance duplication: $\textrm{Sim}(\bm{f}_{i,k},\bm{f}_{i,j})=\frac{\bm{f}_{i,k}^{T}\cdot{\bm{f}_{i,j}}}{\left\|{\bm{f}_{i,k}}\right\|\left\|\bm{f}_{i,j}\right\|}$, where $\bm{f}_{i,k}$ is the feature of the instance $k$ in the image $I_{i}$ extracted by $P_{\mathcal{S}}(\cdot)$. Subsequently, ENMS is performed on the indicating set $S_{ins}$ initialized as $[t]$, where $[t]$ is the set of all instances in $I_{i}$. As summarized in Algorithm 1, the basic idea of ENMS is to select the most informative instance $k_{pick}$ from $S_{ins}$ with the corresponding entropy $\mathbb{H}(I_{i},k_{pick})$ being accumulated for the image-level entropy $E_{i}$. Meanwhile, the remaining within-class instances that are similar to $k_{pick}$ (i.e. the pair-wise similarity is larger than a threshold $T_{enms}$) are deemed as redundant ones, and further removed from $S_{ins}$. The procedure aforementioned is iteratively conducted until $S_{ins}$ becomes empty.

It is worth noting that ENMS only compares instances from the same categories w.r.t. the selected informative instance, and is thus computationally efficient. Meanwhile, ENMS extracts the instance-level features on-the-fly, which significantly reduces the memory cost. Additionally, ENMS can mitigate the unbalanced amount of instances per image, by means of redundant instance removal.

ENMS enhances the intra-image diversity, and the inter-image diversity, i.e. the redundancy across images, still remains. Most conventional approaches [1] address this issue based on holistic image-level features, which are too coarse to fulfill instance-level processing in object detection. Some re-weighting based methods [14, 37] can be adapted from image-level to instance-level, mitigating the inter-image redundancy. However, they need to compute the distances between all instance pairs, which incurs high memory and computational cost. Besides, current studies rarely consider the imbalanced classes of instances, making it hard to estimate the normalized diversity.

Inspired by the previous attempts [31, 41, 35], we introduce the prototypes to address the drawbacks above. Concretely, the $i$-th prototype of class $c$ is formulated as:

where $\mathbbm{1}(c,c_{i,k})$ equals to $1$ if $c=c_{i,k}$, and $0$ otherwise.

As shown in Eq. (3), the prototype is formulated based on the entropy and the predicted class instead of the confidence score as in existing work [35], since our framework focuses on the information gain. Therefore, the instances with high classification confidence scores contribute less to the prototype than the uncertain ones.

Based on ENMS and the prototypes, we propose the Diverse Prototype (DivProto) strategy to enhance the instance-level diversity across images. Specifically, we firstly sort the unlabeled images according to their entropy $\{E_{i}\}$ via ENMS in descending order. Subsequently, we improve the intra-class diversity via intra-class redundancy rejection as shown in Fig. 2 (c) and the inter-class diversity via inter-class balancing as in Fig. 2 (d).

Intra-class Diversity. Given a candidate image $I_{i}$ and the prototypes of the acquired set $\Delta{\mathcal{S}}$, the intra-class diversity of $I_{i}$ is measured by the following metric:

In Eq. (4), we can observe that: 1) by using $M_{g}$, the inter-image diversity is measured by the similarity between the prototypes instead of instance-level pair-wise comparison, thereby remarkably reducing the computational complexity, and 2) $M_{g}(I_{i},[C])$ encodes the similarity between intra-class prototypes across images and increases if $I_{i}$ is more similar to the picked image set $\Delta{\mathcal{S}}$.

Based on the observations above, we thus adopt the following intra-class redundancy rejection process to enhance the intra-class diversity across images: reject the image $I_{i}$ when $M_{g}(I_{i},[C])$ is larger than a threshold $T_{intra}$ (0.7 by default), and accept otherwise.

Inter-class Diversity. Though the intra-class diversity can be enhanced based on $M_{g}(I_{i},[C])$, the image set acquired by the intra-class rejection process tends to favor certain classes (i.e. the majority classes), leading to severe class imbalance. To deal with this issue, we increase the inter-class diversity by introducing the inter-class balancing process, i.e. adaptively providing more budgets for the minority classes than the majority ones.

Concretely, we first build the minority class set $[C_{minor}]$ by sorting the overall classes according to the frequency of occurrences in the labeled image set $\mathcal{S}$ and selecting the classes with the $C_{minor}$ fewest instances, where $C_{minor}=\alpha C$ ($0<\alpha<1$). We assign each minority class $c\in[C_{minor}]$ a relatively large quota $q_{c}=\frac{\beta}{\alpha{C}}b$ ($\alpha<\beta<1$) as the class-specific budget.

For an unlabeled image $I_{i}$, we check if it contains the instances from the minority classes by computing

where $p(i,c)=\max\limits_{k\in[t]}{\mathbbm{1}(c,c_{i,k})\cdot{p_{i,k}}}$ estimates the probability about the existence of instances from the class $c$ in $I_{i}$.

In this work, we adopt a threshold $T_{inter}$ (0.3 by default), where the image $I_{i}$ is accepted as containing minority classes if ${M}_{p}(I_{i},[C_{minor}])>T_{inter}$, and is rejected otherwise. Once $I_{i}$ is accepted, the quota $\{q_{c}\}$ will be updated as $q_{c}:=q_{c}-1$ if $p(i,c)>T_{inter}$. During image acquisition, the class with $q_{c}=0$ will be removed from the minority class set $[C_{minor}]$, while the number of the minority classes $C_{minor}$ is updated by $C_{minor}:=C_{minor}-1$. The whole process is terminated when $C_{minor}$ reaches 0.

Since an image may contain instances from multiple minority classes and $\beta<1$, the number of acquired images by using the inter-class balancing process should not exceed the budget $b$. We therefore fill up $\Delta{\mathcal{S}}$ with the remaining unlabeled images until the budget $b$ runs out.

By performing the process above, we can make a balance w.r.t. the number of instances from various classes, and finally increase the inter-class diversity. The details of DivProto are summarized in Algorithm 2.

Additionally, we report the results with small budgets (no more than 10%) following MIAL [39]. As in Fig. 3 (b), our method still outperforms the counterpart methods by a large margin in those settings. Though the detection performance under such small budgets does not meet the level of real-world applications, the remarkable gains compared with MIAL [39] demonstrate its effectiveness.