Environment-Invariant Curriculum Relation Learning
for Fine-Grained Scene Graph Generation

The scene graph generation task aims to generate a summary graph $\mathcal{G}$ for the given image $I$. Specifically, a scene graph $\mathcal{G}=\left\{\left(O,E\right)\right\}$ corresponding to $I$ contains a set of target entities $O=\left\{\left(o_{i}\right)\right\}_{i=1}^{N_{o}}$ and a set of relational triplets $E=\left\{\left(o_{i},p_{\left(o_{i},o_{j}\right)},o_{j}\right)\right\}_{i,j=1}^{N_{e}}$, where $o_{i}\in O$ and $o_{j}\in O$, $p_{\left(o_{i},o_{j}\right)}$ is defined as the relation between them and belongs to the predefined predicate class set $\mathcal{P}$.

Classifying a relation $p$ as the predicate class $c$ can be preliminarily defined as predicting $P(r=c\mid p)$ based on the dataset of the relations and its label pairs $\{(p,r)\}$ [45]. Using Bayes theorem [3], the predictive model could be decomposed as $P(r=c\mid p)=\frac{P(p\mid r=c)\cdot P(r=c)}{P(p)}$, where $P(r=c)$ is the class distribution, and $P(p)$ is the marginal distribution of the relations. Previous SGG methods [32, 19] only consider class imbalance $P(r=c)$ while ignoring the context imbalance from the different marginal distribution $P(p)$ based on subject-object pairs.

To explicitly define the relation classification $P(r=c\mid p)$, we assume that a relation $p$ is generated by a set of hidden attributes $z=\left\{z_{1},z_{2},z_{3},\ldots\right\}$. Since there exists predicate class imbalance and context imbalance in the existing SGG dataset, we defined two disjoint subsets for the hidden attributes: class-specific attributes $z_{c}$ (e.g., the frequency of the predicate classes [32]) and context-specific attributes $z_{e}$ (e.g., the various context subject-object pairs $\{o_{i},o_{j}\}$). Thus we can further decompose the relation prediction model $P\left(r=c\mid z_{c},z_{e}\right)$ as follows:

From Eq. 1, the relation classifier $P\left(r=c\mid z_{c},z_{e}\right)$ is affected by two imbalances:

Class Imbalance: in previous SGG works [19, 32, 38], the distribution of $P(r=c)$ is considered as the main cause of the performance degradation. As $P(r=c)$ can be explicitly calculated from the training data, the majority of previous methods directly alleviate its effect by class-wise adjustment [38] or re-sampling [19, 12].

Context Imbalance: we argue that the context imbalance suffers the relation classifier in two ways. First, as shown in Fig. 2 (a), the different diversity for the context makes imbalanced influences for the predicate classes. Second, Fig. 2 (b) shows that the predicate is high-related to certain context subject-object pairs. These phenomenons will create spurious correlations between the subject-object pairs and the predicates, which will weaken the relation classification performance, especially in the tail predicates whose context subject-object pairs are rare due to sampling scarcity. Since we have concluded that the context subject-object pair is highly related to the predicate class, we formulate the context imbalance as $P\left(z_{e}\mid r=c,z_{c}\right)$.

To solve these two imbalances and obtain unbiased relations for the scene graphs, we propose the following Environment Invariant Curriculum Relation learning (EICR) framework to learn an unbiased relation classifier invariant to the change of the various predicate classes and context subject-object pairs.

To alleviate the context imbalance, we need to eliminate the impact of the contexts for the relation classification $P\left(z_{e}\mid r=c,z_{c}\right)$. To this end, based on the theory of Invariant Risk Minimization (IRM) [2], we can first construct a set of environments $\mathcal{E}=\left\{e_{1},e_{2},\ldots\right\}$ with diverse context distribution. Then, by regularizing the relation classifier $g(\cdot)$ to be equally optimal across the environments with different context distributions, we can alleviate the influence of the context. Thus objective function can be defined as follows:

where $R^{e}(I,r;f(\cdot),g(\cdot))$ is the risk under environment $e$ (i.e., the loss for training), $f(\cdot)$ is the relation feature extractor, $g\in\arg\min_{g}R^{e}(I,r;f(\cdot),g(\cdot))$ for all $e\in\mathcal{E}$ means that the invariant identifier $g$ should minimize the risk under all environments simultaneously. Following IRM [2], we use a gradient norm penalty term to minimize $g$ at each environment, i.e., $\min_{\Phi:\mathcal{X}\rightarrow\mathcal{Y}}\sum_{e\in\mathcal{E}}R^{e}(\Phi)+\lambda\cdot\left\|\nabla_{w\mid w=1.0}R^{e}(w\cdot\Phi)\right\|^{2}$,where $\Phi$ is the invariant model, $R(\cdot)$ denotes the training loss under different environments $e\in\mathcal{E}$ and we set $\lambda=1$. The detailed process of the environment construction is introduced below:

The set of diverse environments should ensure the variance of the context influence and ideally are orthogonal distributions [34, 43]. However, considering the computation consumption and feasibility of the strategy, to construct different $P\left(z_{e}\mid r=c,z_{c}\right)$, it is hard to change the $r=c$ since the relation labels are predefined. Thus we construct three learning environments with different $z_{c}$, i.e., the frequency of the predicate classes. As illustrated in Fig. 3, each learning environment constructs different frequencies of the predicate classes:

$\bullet$ The normal environment maintains the raw distribution of the predicate classes in the dataset.

$\bullet$ The class-balanced environment constructs the resampling strategy [12] to sampling in all predicate categories at balanced frequencies. Specifically, we first calculate the median amount of the samples over all predicate classes $Median(r)$. Then, for each predicate class $r_{i}$ with $n_{i}$ samples, we calculate the sampling rate $s_{i}$ as follows:

$\bullet$ The over-balanced environment is constructed to over-correct the imbalanced predicate class distribution $P(r)$. Thus we first construct a resampling strategy for the balanced sampling as in the class-balanced environment. Then we adopt an extra reweighting strategy for over-balanced weighting, the loss can be formulated as:

where $w_{i}=1/n_{i}$ and $C$ denotes the number of the predicate categories. This environment deliberately picks relation triplets with the probability negatively correlated with predicate class size.

After obtaining a context-unbiased relation representation from the environment learning module, we assume the network has already modeled the $P\left(z_{e}\mid r=c,z_{c}\right)$. Therefore, we only need to tackle the class imbalance $P(r=c)$ in the context-balanced SGG data. We devise a curriculum schedule for environment learning to make the relation prediction model successfully explore general patterns from head predicates and then gradually focus on the tail predicates. Specifically, we adjust the learning weights between the over-balanced environment and the normal environment by a trade-off factor $\lambda$ which is defined as:

where $t$ is the current training iteration, $T$ is predefined intermediate training iterations for different stages of curriculum learning. $\lambda_{min},\lambda_{max}\in[0,1]$ are hyper-parameters. In order to ensure the scale invariance, $\lambda_{min}+\lambda_{max}=1$. $H(t)$ is a curriculum schedule function decreasing from 1 to 0 with the input iteration $t$, which can be defined as:

thus the joint loss function for the three environments can be formulated as:

where $\mathcal{R}_{norm},\mathcal{R}_{over},\mathcal{R}_{balanced}$ are the risks under normal, class-balanced, and over-balanced environments. The class-balanced curriculum learning strategy thus can be divided into three phases. In the first training phase ($t\leq T$), the model is mainly focused on the normal environment to learn the general patterns from head predicates. In the second phase ($T<t\leq 2T$), $\lambda$ gradually decreases during the training. The model’s learning focus shifts from the normal environment to the over-balanced environment to incrementally learn the fine-grained tail predicates while retaining the general patterns. In the third phase ($t>2T$), the model avoids overfitting the general patterns from the normal environment when focusing on the tail predicates at later training periods. Algorithm.  1 shows details of EICR.

Dataset. In the SGG task, we choose Visual Genome (VG) [17] dataset which comprises 75k object categories and 40k predicate categories. We applied the widely accepted benchmark [45, 33, 32, 47, 28], using the 150 highest frequency objects categories and 50 predicate categories. GQA [15] is another dataset for vision-language tasks with more than 3.8M relation annotations. Following previous work [12], we select Top-200 object classes as well as Top-100 predicate classes by their frequency for the GQA200 benchmark. For both datasets, the training set is set to be 70%, and the testing set is 30%, with 5k images from the training set for validation [45].

Tasks. Following previous works [45, 33, 12, 6], we evaluate our model on three widely used SGG tasks: (1) Predicate Classification (PredCls): given images, object bounding boxes, and object labels, models are required to recognize predicate classes. (2) Scene Graph Classification (SGCls): gives images and object bounding boxes and asks models to predict object labels and relationship labels between objects. (3) Scene Graph Detection (SGDet): models are required to localize objects, recognize objects, and predict their relationships directly from images.

Metrics. Following previous works [19, 38], we use Recall@K (R@K) and mean Recall@K (mR@K) as our metrics. Moreover, inspired by previous work [46], we use the overall metric F@K to jointly evaluate R@K and mR@K, which is the harmonic average of R@K and mR@K.

Implementation Details. We employ a pre-trained Faster-RCNN [29] with ResNeXt-101-FPN [23] provided by [32] as the object detector. We use Glove [27] to obtain the semantic embedding. In the training process, the parameters of the detector are fixed to reduce the computation cost. The hyper-parameter lambda which balances the different environments is set to 0.9. We optimize all models with an Adam optimizer with a momentum of 0.9. The batch size is set to 4, and the total training stage lasts for 120,000 steps with $T=30000$ and $\lambda_{max}=0.9$. The initial learning rate is 0.001, and we adopt the same warm-up and decayed strategy as [12]. One RTX2080 Ti is used to conduct all the experiments.

We demonstrate the effectiveness of our method by comparing the results with current SOTA methods and validate its generalizability with different baseline models. On the one hand, to prove its performance, we select some dedicated designed SGG models with state-of-the-art performance, including re-produced IMP [35], GPS-Net [25], DT2-ACBS [10], SHA-GCL [12], and BGNN [19]. On the other hand, to demonstrate the generalizability of our EICR, we compare our method with the model-agnostic baselines which can be applied in a plug-and-play fashion, including TDE [32], CogTree [44], PCPL [38], EBM [31], DLFE [7], GCL [12], NICE [18] and IETrans [46].

We report the results of the proposed EICR framework and baselines on the VG and GQA datasets in Table  1 and Table  2. From the results of various tasks and baselines, we have several observations as follows:

On the one hand, our EICR is adaptive to different baselines. We adapt our method to 3 popular baselines for SGG, including Motifs [45], VCTree [33], and Transformer [32]. These baselines include various architectures such as conventional LSTM (Motifs), tree structure (VCTree), and self-attention layers (Transformer). Various training algorithms are also contained such as supervised training and reinforcement learning (VCTree). Specifically, our method can boost all models’ mR@50/100 metric and the overall F@50/100 metric. With our method, the results for VCTree are improved over 14% on mR@50/100 and improved over 12% across all 3 tasks on the metric F@50/100. Moreover, compared with other model-agnostic methods, our method outperforms all of them on the F@50/100 and gains competitive results on mR@50/100. By applying our methods to VCTree and Transformer on SGCls and SGDet, our model can achieve the highest R@50/100 and mR@50/100 among all model-agnostic baselines.

On the other hand, compared with strong specific baselines, our method can also achieve competitive performance on mR@50/100 and the best overall performance on F@50/100. Our method with VCTree is close to the SOTA results in DT2-ACBS on SGCls and SGDet tasks on mR@50/100 while outperforming much better than them on R@50/100. For an overall comparison of the F@50/100 metrics, our method with VCTree can achieve the best F@50/100 on PredCls and SGCls and our method with Motif achieves the best F@50/100 in the SGDet task.

In this part, we analyze the influence of the environment-invariant learning, curriculum learning strategy, and corresponding parameter $\lambda_{max}$.

Influence of Environment-Invariant Learning (EIL). Table  3 and Table  4 present the results of all the ablation models. As shown in Table  3, only using environment-invariant learning is hard to boost the mR@50/100 and F@50/100 performance as much as EICR. The reason is that the training procedure is still led by the normal environments and overfitting the corresponding general patterns. However, though the performance on the mR@50/100 and F@50/100 are not boosted so much, the EIL retains the performance on the R@50/100 compared with the original baselines (Motifs, VCTree, Transformer). We can conclude that by introducing EIL to cope with the context imbalance problem, the model learns the context-unbiased relation representation and make a gain on the mR@50/100 metric while retaining the general patterns without the decrease on the R@50/100 metric. As shown in Table  4, different settings of learning environments all achieve improvements on mR@50/100 and F@50/100 compared with Motifs. However, its performance is poor compared with EIL, which shows the importance of combing multiple environments by EIL. Integration of multiple learning environments can alleviate the context imbalance and improve the performance on SGG benchmarks.

Influence of Curriculum Learning Strategy. As aforementioned, we propose the class-balanced curriculum learning strategy to alleviate the class imbalance. In order to prove the effectiveness of the above components, we test various ablation models on the VG dataset as follows:

(1) w/o-Curriculum Schedule: To evaluate the effectiveness of the curriculum schedule, we do not use curriculum schedule, i.e., $\mathcal{R}_{\text{hybird}}=\mathcal{R}_{norm}+\mathcal{R}_{over}+\mathcal{R}_{balanced}$.

(2) w/o-Norm Schedule: To evaluate the effectiveness of the changing weight of the normal environment, we remove the curriculum schedule for the normal environment risk and only employ the curriculum schedule for the over-balanced environment, i.e., $\mathcal{R}_{\text{hybird}}=\mathcal{R}_{norm}+(1-\lambda)\cdot\mathcal{R}_{over}+\mathcal{R}_{balanced}$.

(3) w/o-Over Schedule: To evaluate the effectiveness of the curriculum schedule for the over-balanced environment, we remove the curriculum schedule for the over-balanced environment, i.e., $\mathcal{R}_{\text{hybird}}=\lambda\cdot\mathcal{R}_{norm}+\mathcal{R}_{over}+\mathcal{R}_{balanced}$.

Table  5 presents the results of all the ablation models. First, the curriculum schedule can achieve a huge improvement on the mR@50/100 and F@50/100 metrics. Compared with w/o-curriculum schedule, w-curriculum schedule boosts the mR@50/100 metric by over 5 points and improves by nearly 4 points on the F@50/100. Second, we witness an obvious performance decay when removing the curriculum schedule either for the normal environment or the over-balanced environment. It verifies that constructing curriculum learning schedules for multiple environments would effectively alleviate the class imbalance in the SGG dataset, thus leading to class-unbiased relation predictions.

Influence of IRM regularization. We take Motifs [45] as the baseline model. As shown in Table  3, we can see that adding the IRM regularization term improves the performance of R@50/100 and F@50/100, demonstrating that the IRM regularization enhances the representation ability of the predicate predictor.

Influence of $\lambda_{max}$. As shown in Table  7, the mR@50/100 metric significantly increases with the increase of the $\lambda_{max}$ while the R@50/100 metric decreases at the same time. Since the over-balanced environment is highly related to the samples from the tail predicates, the increase of the $\lambda_{max}$ can somewhat be considered as increasing the model’s attention to the tail samples. Thus, the phenomenon indicates that the conventional structures of the SGG model (Motifs, VCTree, Transformer) may easily classify tail classes as negative samples and lead to low results on mR@50/100, while this part of the data is of vital significance for improving the model’s ability to make class-unbiased predictions.

To get an intuitive perception of the superior performance on the SGG tasks of our proposed method, we make quantitative studies.

Visualization. To show the potential of our method for real-world application, we visualize several PredCls examples generated from the biased Motifs and the unbiased Motifs + EICR. As shown in Fig. 4, we can observe that our method can help to generate more various relation predicates while keeping faithful to the image content. The model prefers to provide more specific relationship predictions (e.g., ‘covering’ and ‘in front of’) rather than common and trivial ones (e.g., ‘at’ and ‘along’). Moreover, our method could also help capture potential reasonable relationships. For example, in Fig. 4, our method captures ‘logo-on-bag’ in the left example and ‘track-in-train’ in the right example. In a nutshell, the proposed method could enhance the unbiased scene graph generation and generate more informative relation triplets to support various downstream tasks.

Detailed Results. In Fig. 5 (a), we show the detailed results of comparing Motifs and Motifs + EICR with respect to R@100 of all the predicate classes on the SGCls task. We can observe that by applying our methods to the Motifs baseline, though there exists an acceptable decay on the minority of several head predicate classes, the performance on most of the predicate classes is obviously improved. Moreover, we also compare Motifs + EIL (i.e., w/o curriculum schedule) and Motifs + EICR on the detailed performance towards every predicate class on VG. As shown in Fig. 5 (b), the curriculum schedule effectively prevents the model from overfitting the general pattern on the head predicate classes and achieves a better performance towards the tail predicate predictions. It demonstrates that the curriculum schedule could achieve a reasonable trade-off between the environments, and effectively alleviate the class imbalance of the predicates to get a class-unbiased relation classifier.

Verification for Alleviating the Context Imbalance. To make a further analysis, we verify the effectiveness of our EICR for alleviating the context imbalance between the various subject-object pairs. Specifically, we calculate the mR@100 of all the different subject categories inside the same predicate class. i.e., the mR@100 for the subject ‘man’ is calculated by the mean R@100 of the relation triplets with the subject ‘man’ such as ‘man-wearing-shirt’ and ‘man-wearing-boots’. Two examples on the VG dataset are shown in Fig. 6. We can see that compared with the Motifs, the EICR help to make a more balanced distribution for the various subject-object pairs thus gaining context-unbiased results.

Discussions with Relevant Long-Tailed Approaches. To demonstrate the effectiveness of our EICR strategy in alleviating the class imbalance, we compare our method with two other relevant typical class-balancing strategies on the Motifs baseline, i.e., the resampling strategy BBN [50] and the reweighting strategy [8] following SHA [12]. As shown in Table 8, we can see that our method achieves the best performance. We can see that compared with the Motifs, our EICR model could help to make a more balanced distribution for the various predicate classes thus gaining better results on the various SGG datasets.

Verification for Balancing Contexts. To provide a more detailed analysis of our method’s effectiveness in alleviating the context imbalance, we report the metric mT@50/100 on the VG dataset. mT@50/100 denotes the average of the mean Recall for various triplets (i.e., the same predicate with different subject-object context) inside each predicate class. As shown in Table  9, our EICR can be applied in a plug-and-play fashion for solving the context imbalance. By adding our EICR to the three baselines, the results are significantly improved across all 3 tasks on the metric mT@50/100. Moreover, in Fig. 7, we show the detailed results of comparing Motifs and Motifs + EICR with respect to mR@100 of the triplets inside all predicate classes on the PredCls task. With our method, the performance of the triplets inside most of the predicate classes is obviously improved. It demonstrates that our method could achieve a reasonable trade-off for the existing imbalance contexts between the predicate classes, and effectively alleviate the context imbalance.

Influence of $T$. To provide a more detailed analysis of the influence of the curriculum learning module, we report the performance with different $T$. As shown in Table  10, with the increase of the intermediate training iterations $T$, the mR@50/100 metric and the overall metric F@50/100 first increases and then decreases. The phenomenon shows that blindly focusing on the tail predicates does not necessarily mean higher performance on the various SGG datasets.