A Study on Self-Supervised Object Detection Pretraining

A large number of recent work on self-supervised learning focuses on constrastive learning, which learns the general feature of an image by using data augmentations and train the model to discriminate views coming from the same image (positive pairs) and other images (negative pairs), usually by using the InfoNCE loss [16, 32, 29]. In practice, contrastive learning requires simultaneous comparison among a large number of sampled images, and benefits from large batches [8, 9], memory banks [17, 10], false negative cancellation [22], or clustering [5, 1, 21]. Recently, non-contrastive methods such as BYOL [15] and its variants [7, 11] obtain competitive results without using negative pairs, by training a student network to predict the representations obtained from a teacher network.

While instance-level representation learning methods show promising results on transfer learning for image classification benchmarks, they are sub-optimal for dense prediction tasks. A recent line of works focus on pre-training a backbone for object detection. Similar to instance-level representation learning, these pre-training methods can be based on contrastive learning (e.g. VADeR [26], DenseCL [30], and PixPro [34]), or self-distillation, (e.g. SoCo [31] and SCRL [28]). These methods share a general idea of leveraging view correspondence, which is available from the spatial relation between augmented views and the original image. Beyond pre-training the backbone, UP-DETR [12] and DETReg [3] propose a way to pre-train a whole object detection pipeline, using a frozen backbone trained with SSL on object-centric images to extract object features. Xie et al. [33] leverage image-level self-supervised pre-training to discover object correspondence and perform object-level representation learning from scene images.

Many of these object detection pre-training techniques rely on heuristic box proposals [31, 3] or frozen backbone trained on ImageNet [12, 3]. Our work, however, aims to study the potential of end-to-end object detection pre-training without them. The framework we study is closest to SCRL [28], which adopts the approach from BYOL [15].

We first randomly crop the original image to obtain a base view, with minimum scale of $s_{\text{base}}$. Next, $V$ augmented views $v_{1},\cdots v_{V}$ are constructed from the base view using $V$ different image augmentations $t_{1},\cdots,t_{V}\sim\mathcal{T}$, each of which is a sequence of random cropping with minimum scale of $s_{\text{view}}$, color jittering, and Gaussian blurring. Here, $\mathcal{T}$ is the distribution of image augmentations. The minimum scale of these $V$ views with regards to the original image is $s_{\text{base}}\times s_{\text{view}}$. We separate $s_{\text{base}}$ and $s_{\text{view}}$ and choose $s_{\text{view}}>0.5$ to make sure that views are pairwise overlapped. The views are also resized to a fixed size to be processed in batches.

Next, we sample $K$ boxes $b^{1},\cdots,b^{K}\sim\mathcal{B}$ relative to the base view, where $b^{k}\in\mathbb{R}^{4}$ is the box coordinate of the top left and the bottom right corner, $\mathcal{B}$ is the box distribution. We transform these boxes from the base view to each augmented view $v_{i}$, based on the transformation $t_{i}$ used to obtain $v_{i}$. We keep only valid boxes that are completely fitted inside the view. Let $\tau_{t_{i}}$ be the box transformation, $B_{i}$ be the set of valid box indices by this transformation, the set of boxes sampled for view $i$ is denoted as $\{b^{k}_{i}=\tau_{t_{i}}(b^{k})|k\in B_{i}\}$. During this sampling process, we make sure that each box in the base view is completely inside at least two augmented views by over-sampling and removing boxes that do not satisfy this requirement.

The result of this view construction and box sampling process is $V$ augmented views of the original image with a set of at most $K$ boxes for each view. Figure 1 describes the view and random box generation with $V=3$ and $K=3$.

Our strategy for training the backbone follows BYOL [15]. We train an online network $f_{\theta}$, whose output can be used to predict the output of a target network $f_{\xi}$, where $\theta$ and $\xi$ are weights of the online and target network, respectively. The target network is built by taking an exponential moving average of the weights of the online network, which is analogous to model ensembling and has been previously shown to improve performance [15, 7].

Specifically, let $y_{i}=f_{\theta}(v_{i}),y^{\prime}_{i}=f_{\xi}(v_{i})$ be the output of the backbone for each view $i$. Different from instance-level representation learning [15, 7] where global representations are compared, we compare local regions of interest refined by sampled boxes. Let $\phi$ be a function that takes a feature map $y$, and box coordinates $b$ to output a box representation $\phi(y,b)$ (e.g., RoIAlign). Following [15], we add projection layers $g_{\theta},g_{\xi}$ for both networks and a prediction layer $q_{\theta}$ for the online network. We obtain box representations $u_{i}^{k}=g_{\theta}(\phi(y_{i},b_{i}^{k}))$ from the online network, and ${u^{\prime}}_{i}^{k}=g_{\xi}(\phi(y^{\prime}_{i},b_{i}^{k}))$ from the target network. The SSL loss is computed as:

Our framework reduces to BYOL when $V=2$, $K=1$, and $\tau_{t_{i}}$ returns the whole view boundary regardless of the transformation $t_{i}$. In this case, only global representation of two augmented views are compared.

The framework is similar to SCRL when $V=2$, $\mathcal{B}$ is a uniformly random box distribution, and $\phi(y,b)$ outputs the $1\times 1$ RoIAlign of the feature map $y$ with regards to box $b$. We do not remove overlapping boxes as in SCRL, since object bounding boxes are not necessarily separated.

In SoCo [31], boxes are sampled from box proposals via the selective search algorithm. A third view (and fourth view), which is a resizing of one of the two views is also included to encourage learning object scale variance. We generalize it to $V$ views in our framework. SoCo offers pretraining FPN layers in Faster R-CNN for better transfer learning efficiency. We however only focus on pretraining the ResNet backbone, which is included in both Faster R-CNN and DETR.

We focus on a general random box sampling strategies, as in [28, 12]. While some box proposal algorithms (e.g., selective search) have been shown to produce sufficiently good object boundaries to improve SSL performance [31, 12], we want to avoid incorporating additional inductive bias in the form of rules to generate boxes, since the efficacy of such rules could depend on the dataset.

We study the effect of four hyperparameters of the random box sampling strategy: (1) number of boxes per image ($K$), (2) box coordinate jittering rate (relative to each coordinate value) ($\%n$), (3) minimum box size ($S_{\text{min}}$), and (4) minimum scale for each view ($s_{\text{view}})$. For each attribute, we report results on several chosen values as in Table 2.

The results are shown in 2. For the number of boxes per image $K$, it can be seen that increasing the number of boxes does not have a large effect on the finetuning performance. The results slightly drop when introducing box jittering, which was proposed in [31]. The approach is pretty robust against changing the minimum box size $S_{\text{min}}$. For the minimum scale for each view $s_{\text{view}}$, it can be observed that having a larger scale (i.e. larger overlapping area between views, which boxes are sampled from) does not help to increase the pretraining efficacy.

We explore three different ways to extract features for each box (choices of $\phi(y,b)$): (1) RoIAlign $1\times 1$ (denoted as ra1), (2) RoIAlign $c\times c$ with crop size $c>1$ (denoted as ra3, ra7, etc.), and (3) averaging cells in the feature map that overlap with the box, similar to $1\times 1$ RoIPooling (denoted as avg). While SCRL and SOCO use ra1 [28, 31], ra7 offers more precise features and is used in Faster R-CNN [27] to extract object features. avg shifts box coordinates slightly, introducing variance in the scale and location.

Additionally, with the use of RoIAlign $c\times c$, we want to examine the necessity of random box sampling. Specifically, we compare the dense features of the shared area of two views, which is similar to comparing $c\times c$ identical boxes forming a grid in the shared area.

The results are shown in Table 3. We observe that ra1 achieves the best performance, although the differences are marginal. Moreover, when not using random box sampling, AP scores drop significantly, hinting that comparing random boxes with diversified locations and scales is necessary for a good pretrained model.

multi-crop has been shown to be an effective strategy in instance-level representation learning [6, 7, 22], with both contrastive and non-contrastive methods. In the context of dense representation learning, the only similar adoption of multi-crop we found is in SoCo [31]; however, their third view is only a resize of one of two main views. We are interested in examining if an adaptation of multi-crop that more closely resembles the original proposal of [6] provides meaningful improvements for our task. We consider two settings for our experiments:

In addition to the SSL loss, we consider a box localization loss to match the objective of SSL pretraining with that of an object detection model. Existing methods usually improve pretraining for dense prediction tasks by leveraging spatial consistency; however, a self-supervised pretext task designed specifically for the object detection tasked has been less studied. UP-DETR [12] has demonstrated that using box features extracted by a well-trained vision backbone to predict box location helps DETR pretraining. In this section, we present our effort to incorporate such object detection pretext tasks into our pretraining framework. Two types of box localization loss $\mathcal{L}_{\text{box}}$ are considered. Given a box feature from a view, we can compute either (1) a box prediction loss, i.e. a contrastive loss that helps predict the corresponding box among up to $K$ boxes from another view; or (2) a box regression loss, an L1 distance and general IoU loss of box coordinates from another view predicted by using a transformer. The final loss is defined as $\mathcal{L}=\mathcal{L}_{\text{BYOL}}+\lambda\mathcal{L}_{\text{box}}$, where $\lambda$ is the weight of the box localization term.