Enhancing Weakly-Supervised Object Detection on Static Images through (Hallucinated) Motion

WSOD. Following [12], the RGB image $\mathbf{I}$ undergoes RoI pooling for each of $R$ visual proposals $v_{i}$ to generate a fixed-length feature vector $\phi(v_{i})$. Detection and classification scores are computed for each proposal and category using parallel fully-connected layers. The ground-truth class label for each image is represented as $y_{c}\in{0,1}$, where $c\in{1,...,C}$ and $C$ is the total number of object categories.

The detection and classification scores are computed:

$$v^{det}_{i,c}=w^{det\intercal}_{c}\phi(v_{i})+b^{det}_{c},\quad v^{cls}_{i,c}=w^{cls\intercal}_{c}\phi(v_{i})+b^{cls}_{c}$$

(1)

where $w$ and $b$ represent the weights and biases. These scores are then transformed into probabilities where $p^{cls}_{i,c}$ denotes the probability of class $c$ being present in proposal $v_{i}$, while $p^{det}_{i,c}$ signifies the probability that $v_{i}$ is crucial for predicting the image-level label $y_{c}$.

$$p^{det}_{i,c}=\dfrac{exp(v^{det}_{i,c})}{\sum_{k=1}^{R}exp(v^{det}_{k,c})},\quad p^{cls}_{i,c}=\dfrac{exp(v^{cls}_{i,c})}{\sum_{k=1}^{C}exp(v^{cls}_{i,k})}$$

(2)

Ultimately, image-level predictions $\hat{p}_{c}$ are calculated and used for training in the absence of region-level labels:

$$\hat{p}_{c}=\sigma\left(\sum_{i=1}^{R}p^{det}_{i,c}p^{cls}_{i,c}\right)$$

(3)

$$\mathcal{L}_{mil}=-\sum_{c=1}^{C}\left[y_{c}\log\hat{p}_{c}+(1-y_{c})\log(1-\hat{p}_{c})\right]$$

(4)

Obtaining motion images. FlowNet 2.0 [7] is employed to compute the optical flow between the image and the successive frame in the video. The resulting optical flow is referred to as GT motion images. Additionally, we leverage Im2Flow [4] to hallucinate motion from static images, thereby obtaining the motion modality in the image dataset. The motion images contain 2 channels, representing the horizontal and vertical components of the optical flow. These 2-channel motion images are converted into RGB images by applying color coding [2], as visualized in Fig. 2. Each distinct color in the resulting images corresponds to a different direction of motion. Additionally, darker colors signify a higher magnitude of motion. This transformation enables them to utilize the same siamese backbone as the RGB images.

Siamese design. In our approach to weakly-supervised object detection, we integrate motion information through a Siamese network [5, 3, 14, 9] employing contrastive learning during training. This strategy allows the utilization of a pre-trained RGB backbone to extract features from both RGB and motion images, without introducing additional complexity to the model. Representation learning is enhanced by introducing a contrastive loss between RGB and motion features. The design of our Siamese network enables the utilization of only RGB images during inference, ensuring no extra overhead on inference time.

As shown in Fig. 1, we extract feature maps from the RGB image $\mathbf{I}$ and the motion image $\mathbf{M}$ using a pre-trained backbone. These feature maps are then passed through adaptive pooling and a Siamese fully connected layer, resulting in projected feature vectors $\psi_{proj}(\mathbf{I})$ and $\psi_{proj}(\mathbf{M})$.

Contrastive learning. The RGB and motion feature vectors, denoted as $\psi_{proj}(\mathbf{I})$ and $\psi_{proj}(\mathbf{M})$ respectively, undergo L2-normalization. Following this normalization, we compute their cosine similarity:

$$S(\mathbf{I},\mathbf{M})=\langle\psi_{proj}(\mathbf{I}),\psi_{proj}(\mathbf{M})\rangle/\rho$$

(5)

where $\rho$, is a learnable temperature parameter.

Given RGB and motion image pairs $(\mathbf{I},\mathbf{M})\in\mathcal{B}$, where $\mathcal{B}$ represents an RGB-motion pair batch, we employ noise contrastive estimation (NCE) [6]. The NCE loss contrasts an RGB image with negative motion images to assess the similarity between the RGB image and its paired motion image, relative to others in the batch:

$\mathcal{L}_{{M}\rightarrow{I}}=-\dfrac{1}{|\mathcal{B}|}\sum_{(\mathbf{I},\mathbf{M})\in\mathcal{B}}log\dfrac{exp(S(\mathbf{I},\mathbf{M}))}{exp(S(\mathbf{I},\mathbf{M}))+\sum_{(\mathbf{I}^{\prime},\mathbf{M}^{\prime})\in\mathcal{B}}exp(S(\mathbf{I},\mathbf{M}^{\prime}))}$

(6)

The second component of the NCE loss, denoted as $\mathcal{L}_{{I}\rightarrow{M}}$, is similarly formulated to contrast a motion image with negative RGB image samples. The average of these two components forms the complete NCE loss:

$$\mathcal{L}_{NCE}=(\mathcal{L}_{{M}\rightarrow{I}}+\mathcal{L}_{{I}\rightarrow{M}})/2$$

(7)

While object motion yields valuable insights into object characteristics and positioning, the presence of camera motion introduces interference, creating noise and hindering the accurate extraction of object motion. Due to the ubiquity of camera motion across images, we conceive an approach to extract camera motion, aiming to enhance the accuracy of object motion extraction.

We make an assumption that the motion values in the corners of an image approximate the background motion. Consequently, we calculate the background motion by taking into account the motion observed in these corners. We first compute per-corner motion values (scalars) by averaging per-pixel values. We then cluster the four corner values into two clusters, and drop any singleton clusters (one corner), to reduce the effect of a corner having any relative mismatch in motion values among the other corners, such as when a part of an object is situated on a corner. Further, we take the average motion of corners to obtain the approximated background motion. Because the background motion is caused by camera motion, we subtract the background motion value from the image to obtain a more accurate object motion estimate.

In Fig. 2, the motion image exhibits a consistent yellow tone across the image, indicating downward camera motion according to the color coding scheme. Our method focuses on the image corners outlined with red rectangles to estimate camera motion. Notably, the exclusion of the right upper corner, affected by object (rather than camera) motion, contributes to the improved accuracy of camera motion approximation. Following the normalization process to tackle camera motion, the depiction of the object “umbrella” illustrates a more accurate representation of its motion.

We choose images in the training set depending on whether there is significant object motion. Following the selection process, the training set include images with more pronounced and meaningful motion, amplifying the potential influence of motion while minimizing noise from low-quality motion and images with limited motion and enhancing detection performance. We employ the selection only on training set as we do not use motion during inference.

The process by which we determine if there is significant object motion involves comparing object and background motion (described next). Thus, it requires that we first estimate which bounding boxes containing objects. Employing ground truth bounding boxes (even just at training time) would violate the WSOD setting, which relies solely on class labels for supervision. To adhere to this setting, we leverage a baseline WSOD method, which serves as a foundation for improvement, generating bounding box predictions for objects in the training set.

Let $ib_{i}$ denote the average magnitude motion value inside the bounding box prediction, where $i\in{1,...,N}$ and $N$ is the total number of images in the training set. On the other hand, $ob_{i}$ represents the average magnitude motion value outside the bounding box prediction. The magnitude motion values are normalized between 0 and 1. Our image selection criterion involves choosing an image if $ib_{i}$ surpasses the minimum motion threshold $m=0.2$ and if the ratio between $ib_{i}$ and $ob_{i}$ exceeds the minimum motion ratio threshold $d=1.5$ ($m$ and $d$ chosen by by subjectively evaluating multiple cases to determine the best fit.). Following this criterion, the variable $s_{i}$ signifies whether an image is selected or not:

$$s_{i}=\begin{cases}1,&\text{if }ib_{i}>m\text{ and }ib_{i}/ob_{i}>d\\ 0,&\text{otherwise}\end{cases}$$

(8)

This ensures objects in training images exhibit sufficient motion overall and compared to the background. Note the YouTube-BB dataset where we apply selection usually features a single object.

In Fig. 3, the image in the top row features a horse, but it is identified as a non-moving object with a magnitude of motion inside box $ib_{1}=0.18$, while the magnitude of motion outside of the box $ob_{1}=0.23$. Thus, our method excludes this image from the WSOD training set due to the weak motion. In contrast, the bus in the image below has substantial motion, as evidenced by a magnitude of motion inside $ib_{2}$ at $0.41$, compared to a magnitude of motion outside $ob_{2}$ at $0.09$. Therefore, this image is selected for training to investigate the impact of motion on the WSOD task.

Common Objects in Context (COCO) [8] is a large-scale image dataset for object detection, segmentation, and captioning. We focus on 18 moving object classes (bird, cat, train, boat, umbrella, motorcycle, elephant, bus, cow, car, horse, bicycle, truck, airplane, giraffe, zebra, dog, bear). We extract hallucinated motion from images to show motion helps object detection of even static images. The training set comprises approximately 32k images, while the test set consists of around 2k images.

YouTube-BB [11] is a video dataset, from which we curate a subset image dataset containing the same classes as those in the COCO dataset by randomly sampling images from videos. GT motion is derived by computing optical flow between each image and its consecutive frame in the video. The training set initially contains about 38k images. Following motion-driven image selection, the training set is refined to approximately 27k images. We employ the same test set utilized in the COCO dataset for our evaluations. The video dataset is preliminary step and proof of concept to reach our ultimate goal (hallucinated motion).

We enhance the MIST [12] baseline by implementing a Siamese network with contrastive learning to incorporate motion modality and improve representation learning on the backbone. The integration of GT motion results in notable improvements, showcasing a 4-10% and 13-15% increase in mAP across various thresholds for YouTube-BB and YouTube-BB /w selection datasets, respectively.

Implementing normalization on GT motion images, aimed at mitigating noise introduced by camera motion, yields improved performance. The GT normalized motion exhibits an advancement over the original GT motion, with a 2-10% and 3-12% increase in mAP for the YouTube-BB and YouTube-BB with selection datasets, respectively.

To more effectively evaluate the influence of motion on improving detection results, we employ a selective approach in image curation from YouTube-BB, as detailed in Sec. 2.3, resulting in the YouTube-BB with selection training set. Due to the substantial reduction in the number of images in training set after the selection process, from 38k to 27k, the results in RGB [12] are worse in YouTube-BB with selection. However, upon incorporating motion in both GT Motion and GT Normalized Motion settings, we observe higher performance improvement over RGB setting, ranging from 13-25%. This improvement surpasses the enhancement observed in the YouTube-BB dataset, which ranged from 4-20%. Thus normalization and selection allow us to observe the influence of motion and empower the proof of concept that motion boosts object detection.

After demonstrating the impact of GT and hallucinated motion on frames from a video dataset, we provide evidence supporting our ultimate goal that hallucinated motion using Im2Flow [4] enhances object detection even in static images. The quality of the hallucination is notably compromised in scenarios involving complex backgrounds, non-moving entities, and small objects. The integration of hallucinated motion with RGB images in the YouTube-BB dataset results in a decrease in performance due to the poor quality of the generated motion. After selecting training images based on the amount of motion in YouTube-BB with selection training set, more reliable hallucinated motion is obtained on the average of training set. Thus we observe a noteworthy improvement of 10-13% in the YouTube-BB with selection dataset when leveraging hallucinated motion.

Ultimately, we apply hallucinated motion to the COCO image dataset where GT motion is unavailable, yielding a performance improvement ranging from 0.4-3%. This underscores the capability of motion to enhance the accuracy of object detection in static images.