Robust Object Detection under Occlusion with
Context-Aware CompositionalNets

CompositionalNets [18] are DCNNs with an inherent robustness to partial occlusion. Their architecture resembles that of a VGG-16 network [29], where the fully connected head is replaced with a differentiable generative compositional model of the feature activations $p(F|y)$ and $y$ is the category of the object. The compositional model is defined as a mixture of von-Mises-Fisher (vMF) distributions:

with $\{\nu_{m}\in\{0,1\},\sum_{m=1}^{M}\nu_{m}=1\}$. Here $M$ is the number of mixtures of compositional models and $\nu_{m}$ is a binary assignment variable that indicates which mixture component is active. $\Theta_{y}=\{\theta^{m}_{y}=\{\mathcal{A}^{m}_{y},\Lambda\}|m=1,\dots,M\}$ are the overall compositional model parameters and $\mathcal{A}^{m}_{y}=\{\mathcal{A}^{m}_{p,y}\}$ are the parameters of the mixture components at every position $p\in\mathcal{P}$ on the 2D lattice of the feature map $F$. In particular, $\mathcal{A}^{m}_{p,y}=\{\alpha^{m}_{p,0,y},\dots,\alpha^{m}_{p,K,y}|\sum_{k=0}^{K}\alpha^{m}_{p,k,y}=1\}$ are the vMF mixture coefficients, $K$ is the number of mixture components and $\Lambda=\{\lambda_{k}=\{\sigma_{k},\mu_{k}\}|k=1,\dots,K\}$ are the parameters of the vMF mixture distributions:

where $Z(\sigma_{k})$ is the normalization constant. The model parameters $\{\Omega,\{\Theta_{y}\}\}$ can be trained end-to-end as described in [18].

Occlusion modeling. Following the approach presented in [17], CompositionalNets can be augmented with an occlusion model. Intuitively, an occlusion model defines a robust likelihood, where at each position $p$ in the image either the object model $p(f_{p}|\mathcal{A}^{m}_{p,y},\Lambda)$ or an occluder model $p(f_{p}|\beta,\Lambda)$ is active:

The binary variables $\mathcal{Z}^{m}=\{z^{m}_{p}\in\{0,1\}|p\in\mathcal{P}\}$ indicate if the object is occluded at position $p$ for mixture component $m$. The occluder model is defined as a mixture model:

where {$\tau_{n}\in\{0,1\},\sum_{n}\tau_{n}=1\}$ indicates which component of the occluder model best explains the data. The parameters of the occluder model $\beta_{n}$ can be learned in an unsupervised manner from clustered features of random natural images that do not contain any object of interest.

A natural way of generalizing CompositionalNets to object detection is to combine them with RPNs. However, our experiments in Section 4.1 show that RPNs cannot reliably localize strongly occluded objects. Figure 2 illustrates this limitation by depicting the detection results of Faster R-CNN trained with CutOut [8] (red box) and a combination of RPN+CompositionalNet (yellow box). We propose to address this limitation by introducing a robust part-based voting mechanism to predict the bounding box of an object based on the visible object parts (green box).

CompositionalNets with detection layer. CompositionalNets as introduced in [18] are part-based object representations. In particular, the object model $p(F|\Theta_{y})$ is decomposed into a mixture of compositional models $p(F|\theta^{m}_{y})$, where each mixture component represents the object class $y$ from a different pose [18]. During inference, each mixture component accumulates votes from part models $p(f_{p}|A_{p,y})$ across different spatial positions $p$ of the feature map $F$. Note that CompositionalNets are learned from images that are cropped based on the bounding box of the object [18]. By making the object centered in the image (see Figure 5), each mixture component $p(F|\theta^{m}_{y})$ can be thought of as accumulating votes from the part models for the object being in the center of the feature map.

Based on this intuition, we generalize CompositionalNets to object detection by introducing a detection layer that accumulates votes for the object center over all positions $p$ in the feature map $F$. In order to achieve this, we propose to compute the object likelihood by scanning. Thus, we shift the feature map w.r.t. the object model along all points $p$ from the 2D lattice of the feature map. This process will generate a spatial likelihood map:

where $F_{p}$ denotes the feature map centered at the position $p$. Using this generalization we can perform object localization by selecting all maxima in $R$ above a threshold $t$ after non-maximum suppression. Our proposed detection layer can be implemented efficiently with modern hardware using convolution-like operations.

Robust bounding box voting. While CompositionalNets can be generalized to localize partially occluded objects using our proposed detection layer, estimating the bounding box of an object under occlusion is more difficult because a significant amount of the object might not be visible (Figure 3). We propose to solve this problem by generalizing the part-based voting mechanism in CompositionalNets to vote for the bounding box corners in addition to the object center. In particular, we learn additional mixture components that model the expected feature activations $F$ around bounding box corners $p(F_{p}|\Theta_{y}^{c})$, where $c=\{ct,bl,tr\}$ are the object center $ct$ and two opposite bounding box corners $\{bl,tr\}$. Figure 3 illustrates the spatial likelihood maps $R^{c}$ of all three models. We generate a bounding box using the two points that have maximal likelihood. Note how the bounding boxes can be localized accurately despite large parts of the object being occluded. We discuss how the parameters of all models can be learned jointly in an end-to-end manner in Section 3.4.

CompositionalNets, as well as standard DCNNs, do not separate the representation of the context from the object. The context can be useful for recognizing objects due to biases, e.g. aeroplanes are often surrounded by blue sky. Relying too strongly on context can be misleading when objects are strongly occluded (Figure 4), since the detection thresholds must be lowered under strong occlusion. This in turn increases the influence of the objects’ context and leads to false-positive detection in regions with no object. Hence, it is important to have control over the influence of contextual cues on the detection result.

In order to gain control over the influence of context, we propose a Context-aware CompositionalNets (CA-CompositionalNets), which separates the representation of the context from the object in the original CompositionalNets by representing the feature map $F$ as a mixture of two models:

Here, $\chi^{m}_{p,y}$ are the parameters of the context model that is defined to be a mixture of vMF likelihoods (Equation 3). The parameter $\omega$ is a prior that controls the trade-off between context and object, which is fixed a priori at test time. Note that setting $\omega=0.5$ retains the original CompositionalNet as proposed in [18]. Figure 4 illustrates the benefits of reducing the influence of the context on the detection result under partial occlusion. The context parameters $\chi^{m}_{p,y}$ and object parameters $\mathcal{A}^{m}_{p,y}$ can be learned from the training data using maximum likelihood estimation. However, this presumes an assignment of the feature vectors $f_{p}$ in the training data to either the context or the object.

Context segmentation. Therefore, we propose to segment the training images into context and object based on the available bounding box annotation. Here, our assumption is that any feature that has a receptive field outside of the scope of the bounding boxes would be considered as a part of the context. We first randomly extract features that are considered to be context into a population during training. Then, we cluster the population using K-means++ algorithm[1] and receive a dictionary of context feature centers $E=\{e_{q}\in\mathbb{R}^{D}|q=1,\dots,Q\}$. We apply a threshold on the cosine similarity $s(E,f_{p})=\max_{q}[(e_{q}^{T}f_{p})/(\left\lVert e_{q}\right\rVert\left\lVert f_{p}\right\rVert)]$ to segment the context and the object in any given training image (Figure 5).

We train our proposed CA-CompositionalNet including the robust bounding box voting mechanism jointly end-to-end using backpropagation. Overall, the trainable parameters of our models are $T^{c}=\{\Omega,\Lambda,\{\Theta^{c}_{y}\},\{\chi^{c}_{y}\}\}$ where $c\in\{ct,bl,tr\}$. The loss function has three main objectives: optimizing the parameters of the generative compositional model such that it can explain the data with maximal likelihood ($\mathcal{L}_{g}$), while also localizing ($\mathcal{L}_{detect}$) and classifying ($\mathcal{L}_{cls}$) the object accurately in the training images. While $\mathcal{L}_{g}$ is learned from images $\hat{I^{c}}$ with feature maps $F^{c}$ that are centered at $c\in\{c,bl,tr\}$, the other losses are learned from unaligned training images $I$ with feature maps $F$.

Training Classification with Regularization. We optimize the parameters jointly using SGD:

where $\mathcal{L}_{class}(y,y^{\prime})$ is the cross-entropy loss between the network output $y^{\prime}=\psi(I,\Omega)$ and the true class label $y$. We use a temperature $\mathcal{T}$ in the softmax classifier: $f(y)_{i}=\frac{e^{y_{i}\cdot\mathcal{T}}}{\Sigma_{i}e^{y_{i}\cdot\mathcal{T}}}$. $\mathcal{L}_{weight}=\left\lVert\Omega\right\rVert^{2}_{2}$ is a weight regularization on the DCNN parameters.

Training the generative context-aware CompositionalNet. The overall loss function for training the parameters of the generative context-aware model is composed of two terms:

In order to avoid the computation of the normalization constants $\{Z[\sigma_{k}]\}$, we assume that the vMF variances $\{\sigma_{k}\}$ are constant. Under this assumption, the vMF parameters $\{\mu_{k}\}$ can be optimized with the loss $\mathcal{L}_{vmf}(F,\Lambda)=\mathcal{C}\sum_{p}\min_{k}\mu_{k}^{T}f_{p}$, where $\mathcal{C}$ is a constant factor [18]. The parameters of the context-aware model $\mathcal{A}^{c}_{y}$ and $\chi^{c}_{y}$ are learned by optimizing the context loss:

where $\pi_{p}\in\{0,1\}$ is a context assignment variable that indicates if a feature vector $f_{p}$ belongs to the context or to the object model. We estimate the context assignments a priori using segmentation as described in Section 3.3. Given the assignments we can optimize the model parameters $\mathcal{A}^{c}_{p,y}$ by minimizing [19]:

The context parameters $\chi^{c}_{p,y}$ can be learned accordingly. Here, $z^{\uparrow}_{p}$ and $m^{\uparrow}$ denote the variables that were inferred in the forward process. Note that the parameters of the occluder model are learned a priori and then fixed.

Training for localization and bounding box localization. We denote the normalized response map of the ground truth class as $X^{c}\in\mathbb{R}^{H\times W}$ and the ground truth annotation as $\bar{X}^{c}\in\mathbb{R}^{H\times W}$. The elements of the response map are computed as:

The ground truth map $\bar{X}^{c}$ is a binary map where the ground truth position is set to $X^{c}(c)=1$ and all other entries are set to zero. The detection loss is then defined as:

End-to-end training. We train all parameters of our model end-to-end with backpropagation. The overall loss function is:

$\epsilon_{1},\epsilon_{2}$ control the trade-off between the loss terms. The optimization process is discussed in more detail in Section 4.

Table 1 shows the results of the tested models on the OccludedVehiclesDetection dataset (see Figure 7 for qualitative results). The models are trained on the images from the original PASCAL3D+ dataset with unoccluded objects.

Faster R-CNN. As we evaluate the performance of the Faster R-CNN, we observe that under low levels of occlusion, the neural network performs well. In mid to high levels of occlusions, however, the neural network fails to detect the objects robustly. When trained with strong data augmentation in terms of partial occlusion using CutOut [8], the detection performance increases under strong occlusion. However, the model still suffers from a $59.9\%$ drop in performance on strong occlusion, compared to the non-occlusion setup. We suspect that the inaccurate prediction is due to two major factors: 1) The Region Proposal Network (RPN) in the Faster R-CNN is not able to predict accurate proposals of objects that are heavily occluded. 2) The VGG-16 classifier cannot successfully classify valid object regions under heavy occlusion.

We proceed to investigate the performance of the region proposals on occluded images. In particular, we replace the VGG-16 classifier in the Faster R-CNN with a standard CompositionalNet classifier [18], which is expected to be more robust to occlusion. From the results in Table 1, we observe two phenomena: 1) In high levels of occlusion, the performance is better than Faster R-CNN. Thus, the CompositionalNet generalizes to heavy occlusions better than the VGG-16 classifier. 2) In low levels of occlusion, the performance is worse than Faster R-CNN. The proposals generated by the RPN seem to be not accurate enough to be correctly classified, as CompositionalNets are high-precision models and require a precise alignment of the bounding box to the object center.

Effect of robust bounding box voting. Our approach of estimating corners of the bounding box substantially improves the performance of the CompositionalNets, in comparison to the RPN. This further validates our conclusion that the CompositionalNet classifier requires precise proposals to classify objects correctly with partial occlusions.

Effect of context-aware representation. With $\omega=0.5$, we observe that the precision of the detection decreases. Furthermore, the performance between $\omega=0.5$ and $\omega=0$ follows a similar trend over all three levels of foreground occlusions: the performance decreases as the level of background occlusion increases from BG-L1 to BG-L3. This further confirms our understanding of the effects of the context as a valuable source of information in object detection.

In the following, we evaluate our model on the OccludedCOCO dataset. As shown in Table 2 and Figure 8, our CA-CompositionalNet with robust bounding box voting outperforms Faster R-CNN and CompNet+RPN significantly. In particular, fully deactivating the context ($\omega=0$) increases the performance compared to the original model ($\omega=0.5$), indicating that too much weight is put on the contextual information in the standard CompNets. Furthermore, controlling the prior of the context model to $\omega=0.2$ reaches an optimal performance under strong occlusion where the context is helpful, but does slightly decrease the performance under low occlusion.