[Re] Don’t Judge an Object by Its Context:
Learning to Overcome Contextual Bias

Singh et al. [9] point out the dangers of contextual bias in visual recognition datasets. They propose two methods, CAM-based and feature-split, that better recognize an object or attribute in the absence of its typical context while maintaining competitive within-context accuracy. To verify their performance, we attempted to reproduce all 12 tables in the original paper, including those in the appendix. We also conducted additional experiments to better understand the proposed methods, including increasing the regularization in CAM-based and removing the weighted loss in feature-split.

As the original code was not made available, we implemented the entire pipeline from scratch in PyTorch 1.7.0. Our implementation is based on the paper and email exchanges with the authors. ¹¹1Our implementation can be found at https://github.com/princetonvisualai/ContextualBias. We spent approximately four months reproducing the paper, with the first two months focused on implementing all 10 methods studied in the paper and the next two months focused on reproducing the experiments in the paper and refining our implementation. Total training times for each method ranged from 35–43 hours on COCO-Stuff [1], 22–29 hours on DeepFashion [7], and 7–8 hours on Animals with Attributes [10] on a single RTX 3090 GPU.

We found that both proposed methods in the original paper help mitigate contextual bias, although for some methods, we could not completely replicate the quantitative results in the paper even after completing an extensive hyperparameter search. For example, on COCO-Stuff, DeepFashion, and UnRel, our feature-split model achieved an increase in accuracy on out-of-context images over the standard baseline, whereas on AwA, we saw a drop in performance. For the proposed CAM-based method, we were able to reproduce the original paper’s results to within 0.5% mAP.

Overall, it was easy to follow the explanation and reasoning of the experiments. The implementation of most (7 of 10) methods was straightforward, especially after we received additional details from the original authors.

Since there was no existing code, we spent considerable time and effort re-implementing the entire pipeline from scratch and making sure that most, if not all, training/evaluation details are true to the experiments in the paper. For several methods, we went through many iterations of experiments until we were confident that our implementation was accurate.

We reached out to the authors several times via email to ask for clarifications and additional implementation details. The authors were very responsive to our questions, and we are extremely grateful for their detailed and timely responses.

According to the original paper, all models use ResNet-50 [4] pre-trained on ImageNet [3] as a backbone and are optimized with stochastic gradient descent (SGD) and a batch size of 200. Each standard model is optimized with an initial learning rate of 0.1, later dropped to 0.01 following a standard step decay process. The input images are randomly resize-cropped to $224{\times}224$ and randomly flipped horizontally during training. We also received additional details from the authors that SGD is used with a momentum of 0.9 and no weight decay. The COCO-Stuff standard model is trained for 100 epochs with the learning rate reduced from 0.1 to 0.01 after epoch 60. The DeepFashion standard model is trained for 50 epochs with the learning rate reduced after epoch 30. The AwA standard model is trained for 20 epochs with the learning rate reduced after epoch 10.

After training with the paper’s hyperparameters, we found that our reproduced standard models for COCO-Stuff and AwA were consistently underperforming against the results in the paper. Thus, we also tried varying the learning rate, weight decay, and the epoch at which the learning rate is dropped to achieve the best possible results. Further details can be found in Appendix C. On both COCO-Stuff and AwA, our hyperparameter search ended up reconfirming the original paper’s hyperparameters as the optimal ones; for DeepFashion, we were able to find an improvement. The original, reproduced and tuned results are shown in Table 1, following explanations of biased categories identification (Section 2.2) and evaluation details (Section 2.3).

The paper identifies the top-20 $(b,c)$ pairs of biased categories for each dataset, where $b$ is the category suffering from contextual bias and $c$ is the associated context category. This identification is crucial as it concretely defines the contextual bias the paper aims to tackle, and influences the training of the "stage 2" models and evaluation of all models.

The paper defines bias between two categories $b$ and $z$ as the ratio between average prediction probabilities of $b$ when it occurs with and without $z$. Note that this definition of bias requires a trained model, unlike the more common definition of bias that only requires co-occurrence counts in a dataset [11]. Following the paper description, we used a standard model trained on an 80-20 split for COCO-Stuff and one trained on the full training set for DeepFashion and AwA. For each category $b$ in a given dataset, we calculated the bias between $b$ and its frequently co-occuring categories, and defined category $c$ as the context category that most biases $b$, i.e. has the highest bias value. Bias is calculated on the 20 split for COCO-Stuff, the validation set for DeepFashion, and the test set for AwA.²²2We received additional information from the original authors that they restricted their COCO-Stuff biased categories to the 80 object categories and performed manual cleaning of the DeepFashion ($b$, $c$) pairs. After the bias calculation, we identified 20 $(b,c)$ pairs with the highest bias values. The paper emphasizes that this definition of bias is directional; it only captures the bias $c$ incurs on $b$ and not the other way around.

We compare our pairs to the paper’s in Tables A1 (COCO-Stuff), A2 (DeepFashion), and  A3 (AwA) in the Appendix. Out of 20 biased categories, 2 of ours differed from the paper’s for COCO-Stuff, 10 differed for DeepFashion, and 2 differed for AwA. The variability is expected, as bias is defined as a ratio of a trained model’s average prediction probabilities which will vary across different models. Nonetheless, we found our pairs to also be reasonable, as our biased categories occur frequently with their context categories and rarely without them. See Appendix B for details.

The paper does not specify image preprocessing or model selection. Following common practice, we resize an image so that its smaller edge is 256 and then apply one of two 224x224 cropping methods: a center-crop or a ten-crop. Both are deterministic procedures. We observed that results with center-crop are consistently better and closer to the paper’s results, hence for all experiments, we report results using center-crop. In our email communications, the authors also specified that they use the model at the end of training as the final model. We confirmed that this is a reasonable model selection method after trying three other selection methods, described in Appendix D.

We emphasize that model evaluation is dependent on the identified biased category pairs. For each $(b,c)$ pair, the test set can be divided into three sets: co-occurring images that contain both $b$ and $c$, exclusive images that contain $b$ but not $c$, and other images that do not contain $b$. Then for each $(b,c)$ pair, the paper constructs two test distributions: 1) the "exclusive" distribution containing exclusive and other images and 2) the "co-occur" distribution containing co-occurring and other images. We suspect that other images are included in both distributions because otherwise, both distributions would have small sizes and only consist of positive images where $b$ occurs, disabling the mAP calculation.

The test distribution sizes can be calculated from the co-occurring and exclusive image counts in Tables A1, A2, A3 in the Appendix. As an example, for the (ski, person) pair in COCO-Stuff, there are 984 co-occuring, 9 exclusive, and 39,511 other images in the test set. Hence, there are ${9+39,511=39,520}$ images in the "exclusive" distribution and ${984+39,511=40,495}$ images in the "co-occur" distribution. For COCO-Stuff, we also report results on the entire test set (40,504 images) for 60 non-biased object categories and for all 171 categories, following the paper.

For COCO-Stuff and AwA, we calculate the average precision (AP) for each biased category $b$, and report the mean AP (mAP) for each test distribution. For DeepFashion, we calculate the per-category top-3 recall and report the mean value for each test distribution. Higher values indicate a better classifier for both metrics.

In Table 1, we report the original, reproduced, and tuned results with the paper’s and our 20 most biased category pairs. Evaluated on the paper’s pairs, our best COCO-Stuff model underperforms the paper’s by 1-3%, our best DeepFashion model outperforms by 2-5%, and our best AwA underperforms on the "co-occur" distribution by 3.2% and matches the "exclusive" distribution within 0.1%. When we evaluate the same models on our biased category pairs, we get similar results for the AwA model, slightly worse results for the DeepFashion model, and significantly worse results for the COCO-Stuff model. Due to this big drop in performance for COCO-Stuff, which we suspect is caused by the discrepancy in the identified biased category pairs, we choose to use the paper’s pairs for training and evaluation in the subsequent sections. Overall, we conclude that the paper’s standard baseline results are reproducible as we were able to train models within a reasonable margin of error.

The CAM-based method operates on the following premise: as $b$ almost always co-occurs with $c$, the network may learn to inadvertently rely on pixels corresponding to $c$ to predict $b$. The paper hypothesizes that one way to overcome this issue is to explicitly force the network to rely less on $c$’s pixel regions. This method uses class activation maps (CAMs) [12] as a proxy for object localization information. For an image $I$ and category $r$, CAM($I$, $r$) indicates the discriminative image regions used by a deep network to identify $r$. For each biased category pair ($b$, $c$), a minimal overlap of their CAMs is enforced via the loss term:

$$L_{O}=\textstyle\sum_{I\in\mathbb{I}_{b}\cap\mathbb{I}_{c}}\text{CAM}(I,b)\odot\text{CAM}(I,c),$$

(1)

where $\odot$ denotes element-wise multiplication and $\mathbb{I}_{b}\cap\mathbb{I}_{c}$ is a set of images where both $b$ and $c$ appear. To prevent a trivial solution where the CAMs of $b$ and $c$ drift apart from the actual pixel regions, the paper uses a regularization term to keep the category’s CAMs close to CAM${}_{\text{pre}}$, produced using a separate network trained offline:

$$L_{R}=\textstyle\sum_{I\in\mathbb{I}_{b}\cap\mathbb{I}_{c}}|\text{CAM}_{\text{pre}}(I,b)-\text{CAM}(I,b)|+|\text{CAM}_{\text{pre}}(I,c)-\text{CAM}(I,c)|.$$

(2)

In our implementation, we separate a batch into two small batches during training, one with and one without co-occurrences. A sample is put into the co-occurrence batch if any of the 20 biased categories co-occurs with its context. For the co-occurrence batch, we compute CAM with the current model being trained and CAM${}_{\text{pre}}$ with the trained standard model, using the official CAM implementation: https://github.com/zhoubolei/CAM. We update the model parameters with the following loss, where $L_{\text{BCE}}$ is the binary cross entropy loss:

$$L_{\text{CAM}}=\lambda_{1}L_{O}+\lambda_{2}L_{R}+L_{\text{BCE}}.$$

(3)

For the other batch without any co-occurrences, we update the model parameters with $L_{\text{BCE}}$. With the hyperparameters reported in the paper, $\lambda_{1}=0.1$ and $\lambda_{2}=0.01$, we got underwhelming results and degenerate CAMs that drifted far from the actual pixel regions. Hence, we tried increasing the regularization weight $\lambda_{2}$ (0.01, 0.05, 0.1, 0.5, 1.0, 5.0) and achieved the best results with $\lambda_{2}=0.1$, which are reported in Table 2.

By discouraging mutual spatial overlap, the CAM-based approach may not be able to leverage useful information from the pixel regions surrounding the context. Thus, the paper proposes a second method that splits the feature space into two subspaces to separately represent category and context, while posing no constraints on their spatial extents. Specifically, they propose using a dedicated feature subspace to learn examples of biased categories appearing without their context.

Given a deep neural network, let $\mathrm{x}$ denote the $D$-dimensional output of the final pooling layer just before the fully-connected (fc) layer. Let the weight matrix associated with fc layer be $W\in R^{D\times M}$, where $M$ denotes the number of categories given a multi-label dataset. The predicted scores inferred by a classifier (ignoring the bias term) are $\hat{y}=W^{T}\mathrm{x}$. To separate the feature representations of a biased category from its context, the paper does a random row-wise split of $W$ into two disjoint subsets: $W_{o}$ and $W_{s}$ (dimension $\frac{D}{2}\times M$).³³3In an email, the authors noted that a random split is not critical; they obtained similar results with a random split and a middle split. We observed that a middle split of $W$ yields better results for COCO-Stuff and DeepFashion, but the opposite for AwA. As the gains from using a middle split for COCO-Stuff and DeepFashion were larger than the losses for AwA, we chose to use a middle split. Consequently, $\mathrm{x}$ is split into $\mathrm{x}_{o}$ and $\mathrm{x}_{s}$, and $\hat{y}=W^{T}_{o}\mathrm{x}_{o}+W^{T}_{s}\mathrm{x}_{s}$. When a biased category occurs without its context, the paper disables backpropagation through $W_{s}$, forcing the network to learn only through $W_{o}$, and set $\mathrm{x}_{s}$ to $\bar{\mathrm{x}}_{s}$ (the average of $\mathrm{x}_{s}$ over the last 10 mini-batches).

We implemented the feature-split method based on additional discussions with the original authors, to ensure that we replicated their method as closely as possible. For a single training batch, we first forwarded the entire batch through the model to obtain one set of scores $\hat{y}_{\text{non-exclusive}}=W^{T}_{o}\mathrm{x}_{o}+W^{T}_{s}\mathrm{x}_{s}$ and the corresponding features from the avgpool layer, which directly precedes the fc layer. We made a separate copy of these features and replaced $\mathrm{x}_{s}$ with $\bar{\mathrm{x}}_{s}$, then calculated a new set of output scores $\hat{y}_{\text{exclusive}}=W^{T}_{o}\mathrm{x}_{o}+W^{T}_{s}\bar{\mathrm{x}}_{s}$. Separate loss tensors for each of these outputs were computed, and elements corresponding to the exclusive and non-exclusive examples in the unmodified and modified loss tensors were zeroed out, respectively. The final loss tensor was obtained by adding these two together, and standard backpropagation was done using this final loss tensor. The gradients were calculated with respect to a weighed binary cross entropy loss:

$$L_{\text{WBCE}}=-\alpha\big{[}t\log(\sigma(\hat{y}))+(1-t)\log(1-\sigma(\hat{y}))\big{]},$$

(4)

where $t$ is the ground-truth label, $\sigma$ is the sigmoid function, and $\alpha$ is the ratio between the number of training images in which a biased category occurs in the presence of its context and the number of images in which it occurs in the absence of its context. A higher value of $\alpha$ indicates more data skewness.⁴⁴4In practice, the paper ensures $\alpha$ is at least $\alpha_{\text{min}}$, which they set to 3 for COCO-Stuff and AwA and 5 for DeepFashion. However, we found that most of the paper’s biased category pairs have $\alpha$ smaller than $\alpha_{\text{min}}$. Out of 20 pairs, 13 pairs for COCO-Stuff, 20 pairs for DeepFashion, and 19 pairs for AwA had $\alpha$ smaller than $\alpha_{\text{min}}$. We also tried using higher values of $\alpha_{\text{min}}$ but didn’t gain meaningful improvements, so we report results with the original authors’ $\alpha_{\text{min}}$.

In addition to the standard model, the paper compares the proposed methods with several competitive strong baselines.

1. 1.

   Remove co-occur labels: For each $b$, remove the $c$ label for images in which $b$ and $c$ co-occur.

2. 2.

   Remove co-occur images: Remove training instances where any $b$ and $c$ co-occur. For COCO-Stuff, this process removes 29,332 images and leaves 53,451 images in the training set.

3. 3.

   Split-biased: Split each $b$ into two classes: 1) $b\setminus c$ and 2) $b\cap c$. Unlike other "stage 2" models, this model is trained from scratch rather than on top of the standard baseline because it has 20 additional classes. We later confirmed with the authors that they did the same.

4. 4.

   Weighted loss: For each $b$, apply 10 times higher weight to the loss for class $b$ when $b$ occurs exclusively.

5. 5.

   Negative penalty: For each $(b,c)$, apply a large negative penalty to the loss for class $c$ when $b$ occurs exclusively. In our email communication, the authors said that the negative penalty means a 10 times higher weight to the loss.

6. 6.

   Class-balancing loss [2]: For each $b$, put the images in three groups: exclusive, co-occurring, and other. The weight for each group is $(1-\beta)/(1-\beta^{n})$ where $n$ is the number of images for each group and $\beta$ is a hyperparameter. The authors said they set $\beta=0.99$ in our email communication.

7. 7.

   Attribute decorrelation [5]: Use the proposed method, but replace the hand-crafted features used in  [5] with deep network features (i.e., conv5 features of a trained “stage 1" ResNet-50).

We trained all "stage 2" models on top of the standard model for 20 epochs using a learning rate of 0.01, a batch size of 200, and SGD with 0.9 momentum. The exceptions are split-biased which is not trained on top of the standard model and is thus trained for an additional 20 epochs to ensure a fair comparison; and CAM-based which uses a batch size of 100 due to memory limits. All models were trained on a single RTX 3090 GPU and evaluated on the last epoch. On COCO-Stuff, the single-epoch training time was around 12.9 minutes for standard, remove labels, split-biased, weighted, negative penalty, and class-balancing. It took 8.4 minutes to train remove images, and 17.3 minutes and 13.3 minutes to train CAM-based and feature-split, respectively. Thus, we reach a different conclusion from the paper’s claim that the "overall training time of both proposed methods is very close to that of a standard classifier." We suspect that this difference is due to the difference in implementation. Overall, the total training time for each method range from 35-43 hours on COCO-Stuff, 22-29 hours on DeepFashion, and 7-8 hours on AwA. For inference, the paper reports that a single forward pass of an image takes 0.2ms on a single Titan X GPU for the standard, CAM-based, and feature-split methods. We confirmed that it takes the same amount of time for the three methods. See Appendix E for detailed training and inference times.

In Table 2, we compare the performance of the ten methods, evaluated with the paper’s biased category pairs for consistency. Additiional figures and per-category results can be found in Appendix F and  H.

In Table 3, we compare our reproduced results with the paper’s results. Consistent with the paper’s conclusion, we find that the proposed methods have weights with similar or lower cosine similarity. On the interpretation of the results, we agree that feature-split’s low cosine similarity suggests that the corresponding feature subspaces $\mathrm{x}_{o}$ and $\mathrm{x}_{s}$ capture different information, as intended by the method. However, we don’t understand why the cosine similarity of CAM-based would be lower than standard, as there is nothing in CAM-based that encourages the feature subspaces to be distinct.