Transformer-based Dual Relation Graph for Multi-label Image Recognition

As aforementioned, one crucial problem in multi-label recognition is to capture long-term contextual information and build structural interactions between different objects. Due to the intrinsic flaws of CNN-based architectures, the position correlations are locally conducted without perceiving global contextual information. To mitigate this issue, we make the first attempt to introduce Transformer into multi-label recognition tasks to capture long-term contextual information and build position-wise spatial relationships.

Revisiting Transformer for Structural Relation. In the field of natural language processing, the conventional transformers [31] take language sentences as input and build relationships between different semantic words from global perspectives. Different from language processing, images cannot be directly converted into sequence form. Hence, there are two popular ways to apply transformer on images, embedding transformer into CNN backbones [1] and applying transformer on sequential embedding features of image patches [11]. The latter leads to a high computation burden for network optimization with limited data. Hence in our framework, we adopt the former scheme to capture global contextual information.

We apply standard transformer encoder structure as transformer unit $\mathcal{G}^{trans}_{i}$ to build long-term relationships between pair-wise positions. As shown in Fig. 3, each transformer unit consists of $n$ groups multi-head self-attention modules and feed forward networks, which is composed of two linear transformation layers. The detailed structure of multi-head self-attention module is illustrated in Fig. 3. For each head, we first adopt relative positional encoding $\mathcal{E}(\cdot)$ on the channel-wise transformed feature $\phi(\bm{\mathrm{X}})$ to keep the position information:

where $\mathcal{R}(\cdot)$ denotes the reshape operation, which squeezes the spatial dimensions into one dimension. Then we respectively obtain query, key and value projections of encoded features by linear transformation layers. To build and enhance global position relationships, we calculate the positional correlation matrix $\bm{\mathrm{A}}^{p}$ by query and key, and reweight value with $\bm{\mathrm{A}}^{p}$ by multiplication:

where $\bm{\mathrm{W}}_{\{Q,K,V\}}$ are the learnable weights of query, key and value projections, $\bm{\mathrm{H}}$ is the enhanced feature of one head. Different heads could mine different structural relationships due to different projections. Hence employing multiple heads could capture more comprehensive structural relationships to enrich the representations. For multi heads, we concatenate results from multi heads and fuse them with a linear transformation layer.

Cross-scale Attention Transformers. Small objects tend to have lower performance in multi-label recognition due to the position information of small objects may lost in low-resolution features especially for challenging datasets, e.g., MS-COCO. To address this issue, a natural idea is to consider more high-resolution features to retain the position information of small objects. In fact, high-resolution features indeed improve the performance of small objects, but also introduce more computation burden and noise, which lower the performance of other objects. Towards this concern, we propose a simple yet effective cross-scale attention module as a trade-off between performance and computation cost, which effectively improves the capacity of our structural relation graph.

To suppress noise between different scales and enhance the structure information of small objects, we propose cross-attention feature fusion strategy $\Psi_{i}(\cdot)$. We extract the common positions while alleviating the ambiguous ones by position-wise multiplication operation after up-sampling features of different scales. To enhance the positional information, the extracted feature is respectively down-sampled and enhanced with position-wise addition operation. Thus the structural feature $\bm{\mathrm{T}}_{i}$ of $i$th scale can be formed by this serialized operation:

where $\mathcal{U}(\cdot)$ and $\mathcal{D}(\cdot)$ denote up- and down-sampling operation. Then we input enhanced features into weight-sharing transformer unit to respectively capture the structural relationships with different scales, and the final feature $\bm{\mathrm{T}}$ in Eq. (1) can be obtained by concatenating each $\bm{\mathrm{T}}_{i}$.

Motivation and Discussions. Motivated by the co-occurrence dependencies in multi-label learning, existing works usually resort to graph networks to model this relationship into deep CNNs. Pioneer works [8] in Fig. 4 a) tend to build static correlations of different linguistic word embedding from statistic priors. However, in this label graph, the characteristic of each image are less taken into consideration, which would lead to the hallucination of nonexistent objects and suppression of less common co-occurrences. To this end, in Fig. 4 b), semantic graphs in [5] are built to incorporate high-level features into word embedding. Despite the additional dependencies on word embedding and dataset statistics, high-level features only present implicit semantics for graph construction and detailed relationship between multiple objects are still neglected.

To revisit the graph construction of multi-label learning, here we propose a joint relation graph in Fig. 4 c), involving two meaningful cues for object relation discovery: 1) introducing explicit semantic-aware high-level features via an auxiliary semantic-aware constraint; 2) incorporating structural relationship for graph nodes as well as correlations. The former cue helps to construct an explicit relationship of semantic classes while the latter cue provides the graph a spatial awareness of contextual objects.

Semantic-aware Constraints. Different from previous researches using high-level features $\bm{\mathrm{X}}$ for implicit semantic, here we introduce the explicit embedding with class-specific vectors $\bm{\mathrm{M}}=\phi_{m}(\bm{\mathrm{X}})\in\mathbb{R}^{N_{cls}\times H\times W}$, which is regularized by explicit classification constraints. $\phi_{m}$ denotes the learnable $1\times 1$ convolutional layer. Hence we conduct a high-order fusion to form the semantic-aware features $\bm{\mathrm{V}}_{G}$:

where $\phi_{g}(\cdot)$ denotes a dimension-reduction operation from $C$ to $C_{G}$. However, how to ensure the representation quality of $\bm{\mathrm{V}}_{G}$ for each class is less explored, which affects the subsequent modeling process as an important precondition. To solve this issue, we adopt a global pooling operation, i.e. top-k max-pooling on $\bm{\mathrm{M}}$ to squeeze spatial dimensions and then apply an auxiliary loss on $\bm{\texttt{KMP}}(\bm{\mathrm{M}})\in\mathbb{R}^{N_{cls}}$ to constraint $\bm{\mathrm{M}}$ for learning more accurate initial activation maps and less noise for each class.

Besides the semantic-aware vectors, we also introduce structural information from structural relation graphs to incorporate long-term contextual information and position-wise relationships:

where $\bm{\mathrm{V}}_{T}$ denotes the structure-aware vectors and GAP denotes the global average pooling operation.

Joint Relation Graph. Graph neural networks propagate messages between adjacent nodes based on correlation matrix. As in Fig. 4 c), we build the joint correlation matrix $\bm{\mathrm{A}}^{s}$ from two aspects, i.e. semantic correlations $\bm{\mathrm{V}}_{G}$ and structural correlations $\bm{\mathrm{V}}_{T}$ in a learnable manner:

where $\varphi_{\{c,t\}}$ denote the learnable dimension transformation operation, e.g., $1\times 1$ convolutional layer.

Obtaining graph nodes $\bm{\mathrm{V}}$ = $\bm{\texttt{concat}}(\bm{\mathrm{V}}_{G},\bm{\mathrm{V}}_{T})$ and correlation matrix $\bm{\mathrm{A}}^{s}$, we further model the joint co-occurrence dependencies between joint structural and semantic-aware vectors based on correlation matrix using Kipf et al.’s [20] Graph Convolutional Networks, which can be formulated as:

where $\bm{\mathrm{G}}$ denotes the updated semantic relation graph, $\bm{\mathrm{W}}_{G}\in\mathbb{R}^{(C_{G}+C_{T})\times(C_{G}+C_{T})}$ is the learnable graph weights. $\delta(\cdot)$ denotes the LeakyReLU [26] activation function.

With structural and semantic relation graphs obtained, we further joint their predictions for training in a collaborative learning manner (see Fig. 2). We adopt $\mathcal{L}_{sac}$ to constraint semantic-aware features in Section 3.2. To accelerate the convergence process, we adopt $\mathcal{L}_{trans}$ and $\mathcal{L}_{gcn}$ on the prediction result of structural and semantic relation graphs respectively. Besides, we adopt $\mathcal{L}_{joint}$ for the final prediction result. All these loss functions are supervised with typical multi-label classification entropy $\mathcal{L}_{joint}$ = $-\sum_{i=1}^{N_{cls}}\bm{\mathrm{y}}_{i}\log(\bm{\mathrm{p}}_{i}),\bm{\mathrm{p}},\bm{\mathrm{y}}\in\mathbb{R}^{N_{cls}},\bm{\mathrm{y}}\in\{0,1\}$. Hence the final learning objective $\mathcal{L}_{sum}$ can be formulated as:

With this collaborative regularization, the final classification embedding in Eq. (3) can jointly be aware of structural and semantic information for multi-label understanding.

MS-COCO Benchmark. Microsoft COCO [23] is a widely-used benchmark for many vision tasks, e.g. object detection, segmentation and multi-label recognition. It contains 82,081 images in train set and 40,137 images in validation set from 80 common object categories. On average, each image has 2.9 labels. Especially, it contains a large number of small objects, which is more challenging for multi-label recognition. Following [8, 41, 5], we evaluate the performance of all methods on the validation set.

VOC 2007 Benchmark. PASCAL VOC 2007 [13] is another popular benchmark for multi-label recognition. It contains 5,011 images in train and validation set and 4,952 images in test set from 20 common object categories. On average, each image has 1.4 labels. Following [8], we train our approach on the trainval set and evaluate on the test set.

Evaluation Metrics. To quantitatively evaluate the performance of our approach and state-of-the-art methods, we adopt the average per-class precision (CP), recall (CR), F1 (CF1), the average overall precision (OP), recall (OR), F1 (OF1) and mean average precision (mAP) as evaluation metrics. For fair comparisons, we also report top-3 results. Notably, precision/recall/F1-score may be affected by the threshold, which is set as 0.5 in our setting. Among all metrics, AP and mAP are the most important metrics which can provide a more comprehensive comparison.

We adopt ResNet-101 [17] pre-trained on ImageNet [9] as our backbone. In the training stage, input images are first resized into 512$\times$512, then random cropped and resized into 448 $\times$ 448 with random horizontal flips for augmentation. In the testing stage, input images are resized into 448 $\times$ 448. We use the SGD optimizer with momentum of 0.9 and weight decay of 1e-4. The initial learning rate is 0.01 for VOC 2007 and 0.03 for MS-COCO, which decays by a factor of 10 for every 30 epochs. The batch size is set as 16 for VOC 2007 and 32 for MS-COCO on each GPU. The network converge quickly and only needs 50 epochs in total for training. Detailed experiments on hyper-parameters can be found in supplementary materials. We set hidden dimension $C_{T}=512$ and $C_{G}=512$. The transformer unit consists of 3 layers and each layer has 4 attention heads. The semantic graph neural network has one layer. As mentioned in Eq. (3), We apply a weight coefficient $\alpha$ on structural relation graph and $(1-\alpha)$ on semantic relation graph. We set $\alpha=0.7$ to achieve the best performance. All experiments are conducted on two NVIDIA Tesla V100 GPUs.

Comparisons on MS-COCO. As shown in Tab. 1, we compare our approach on MS-COCO benchmark with 11 state-of-the-art methods. The most commonly used resolution is 448$\times$448 during both training and testing stage. However, it is worthy to notice that some methods evaluate their performance on different resolutions during training and inference stages, e.g., ADD-GCN [41] and SSGRL [5]. For fair comparisons, we follow their resolution settings [8, 41, 5] and report three results, which achieve a new performance leader-board with a clear margin.

Comparisons on VOC 2007. In Tab. 2, we compare our approach with 8 state-of-the-art methods. For fair comparisons, we report mAP and AP of each class on commonly used 448$\times$448 resolution with only ImageNet pretrained. In terms of mAP, our approach achieves the best performance and outperforms state-of-the-art ML-GCN [8] by 1.0%.

Ablation Studies. To evaluate the effectiveness of our proposed structural relation graph module and semantic relation graph module, we reconstruct our model with different ablation factors in Tab. 3. We first employ ResNet-101 with the identical training protocol as our baseline model in the first row, which reaches a high baseline performance, e.g., 92.9% on VOC 2007. Note that this baseline model outperforms several state-of-the-art models and our proposed model can steadily improve the performance based on this high baseline. As shown in Tab. 3, our proposed modules make a steady improvement to the final performance, which demonstrates the necessity of the proposed modules to obtain the best classification results.

Effects of Structural Relation. It can be found in Tab. 3 that only adopting Transformer for structural relation can notably improve the performance, e.g., 4.3% on MS-COCO, which demonstrates the effectiveness of long-term contextual information for multi-label recognition task. Besides, the position information of small objects may vanish after down-sampling especially for challenging datasets, e.g., MS-COCO. Our proposed cross-scale attention module could enhance cross-scale features and suppress noises, which further boosts performance by 1.0% on MS-COCO.

To verify the effectiveness of our proposed cross-scale attention module, we explore different scales on MS-COCO as shown in Tab. 4. The default scale is $\frac{1}{32}$ generated by the last stage of our baseline ResNet-101. With more different scales taken into consideration, the performance of structural relation graph module is slightly improved. With our proposed cross attention module with multiplication operation, the performance further boosts to a new level, the performance drops 0.7% when replacing the multiplication with summation, which demonstrates our proposed cross attention module could further effectively strengthen the position information between different scales.

To further verify the effectiveness of transformer units on cross-scale information, we replace transformer units with a simple MLP layer in the 5th row in Tab. 4, the performance of $R_{structural}$ shows a clear drop (0.6%) in mAP, which demonstrates the transformer units could effectively capture the long-term spatial context.

Effects of Semantic Relation. As shown in Tab. 3, building semantic relationships with GCN could notably improve the performance, e.g. 3.9% on MS-COCO. Moreover, the performance further boosts for 1% with our proposed semantic-aware constraints, which demonstrates that GCN could achieve better modeling results with more representative semantic-specific vectors. To evaluate the effectiveness of our proposed semantic relation graph module, we conduct detailed ablations on MS-COCO in Tab. 5. We adopt static adjacent matrix used in ML-GCN [8] to form a baseline in the first row. Applying our proposed learnable correlation matrix improves the performance by 0.6%. Besides, joint semantic and structural information could effectively improve the performance by 0.5% with structural guidance. Another main exploration is to find global pooling operations for semantic-aware constraints, our final semantic relation achieves the best performance with top-k max-pooling operation with threshold 5%.

Interpretable Visualizations of Structural Relation. We utilize Grad-CAM [28] to exhibit the visualization results of baseline and the proposed structural relation in Fig. 5. i) Benefiting from the cross-scale attention module, our approach could capture more accurate localization and effectively perceive small objects, e.g. fork in Fig. 5 a) and spoon in Fig. 5 b). ii) As baseline model could not distinguish objects with similar appearances, e.g. the triplet labels {fork, knife, spoon} in Fig. 5 c) and the paired labels {backpack, handbag}, {skateboard, snowboard} in Fig. 5 d), these issues are well handled by our proposed structural relation module benefiting from the long-term contextual information captured by Transformer-based relation graph.