GINet: Graph Interaction Network for Scene Parsing

Different from previous methods that only perform contextual reasoning over the visual graph built on visual features [25, 10], our GINet facilitates the graph reasoning by incorporating semantic knowledge to enhance the visual representations. The proposed framework is illustrated in Figure 2. Firstly, we adopted a pre-trained ResNet [20] as the backbone network, where visual features can be extracted given an input 2D image. Meanwhile, the dataset-based linguistic knowledge can be extracted in the form of categorical entities (classes), which is fed into word embedding (e.g., GloVe [35]) to achieve semantic representations. Secondly, visual features and semantic embedding representations are passed by the graph projection operations in the proposed GI unit to construct two graphs, respectively. A detailed definitions of graph projection operations are presented in Section 3.2. Accordingly, one graph that encodes dependencies between visual areas is built over visual features, where nodes indicate visual regions and edges represent the similarity or relation between those regions. The other graph is built over the dataset-dependent categories (represented by word embeddings), which encodes the linguistic correlation and label dependency. Next, a graph interaction operation is processed in the GI unit, where the semantic graph is employed to promote contextual reasoning over the visual graph and guide the generation of the exemplar-based semantic graph extracted from the visual graph. Then, the evolved visual graph generated by the GI unit is passed by Graph Re-projection operation for enhancing the discriminative ability for each local visual representation, while the semantic graph is updated and constrained by the Semantic Context loss during the training phase. Finally, we employ an $1\times 1$ Conv followed by a simple bilinear upsampling to obtain the parsing results.

The goal of the proposed GI unit is to incorporate dataset-based linguistic knowledge for promoting the local representations. First, the GI unit takes visual and semantic representations as inputs, conduct contextual reasoning by generating a visual graph and a semantic graph. Second, a graph interaction is performed between the two graphs to evolve node features by the guidance of similarity of visual nodes and semantic nodes.

Graph Construction:

The first step is to define the projection that maps the original visual and semantic features to an interaction space. Formally, given the visual feature maps $X\in\mathbb{R}^{L\times C}$, where $L=H\times W$, $H$ and $W$ indicate the height and width of the feature map, and $C$ is the channel dimensions. We aim to construct a visual graph representation $P\in\mathbb{R}^{N\times D}$, where $N$ is the number of nodes in the visual graph and $D$ is the desired feature dimension for each node. Inspired by the works [26, 17], we introduce a transformation matrix $Z\in\mathbb{R}^{N\times L}$ that projects the local presentation $X$ to a high-level graph representation $P$, which can be computed as follows:

where $W\in\mathbb{R}^{C\times D}$ is introduced as trainable parameters to convert the feature dimension from $C$ to $D$, and $Z$ adaptively aggregates local features to a node in the visual graph.

Next, we define a dataset-dependent semantic graph. Particularly, we aim to build a semantic graph presentation $S\in\mathbb{R}^{M\times D}$ over the object categories of a specific dataset, to encode the linguistic correlation and label dependency. $M$ denotes the number of nodes, which is equal to the number of categorical entities (classes) in the dataset. $D$ is the feature dimension for each node in the semantic graph. Specifically, we first use the off-the-shelf word vectors [35] to get semantic representation $l_{i}\in\mathbb{R}^{K}$ for each category $i\in\{0,1,..,M-1\}$, $K=300$. Then, a MLP layer is employed to adjust the linguistic embedding to suit the reasoning with the visual graph. This transformation process can be formulated as follow:

where $S_{i}$ represents node features for each category.

Graph Interaction: Next, we present how our GI unit incorporates dataset-based linguistic knowledge for promoting context reasoning and extracting the exemplar-based semantic graph from the visual graph. For simplicity, we abbreviate the visual graph and semantic graph as VisG and SemG, respectively. We first evolve both graphs separately and then perform the interaction between graphs. Then SemG and VisG attentively propagate information interatively, including 1) Semantic to Visual (S2V), and 2) Visual to Semantic (V2S).

Specifically, we first perform graph convolution [23] on the VisG to get evolved graph representation $\widetilde{P}$ that is suitable for interacting with the SemG. This process can be formulated as follows:

where the adjacency matrix $A_{v}\in\mathbb{R}^{N\times N}$ is randomly initialized and updated by the gradient descent; $I$ is an identity matrix; $W_{v}\in\mathbb{R}^{D\times\ D}$ are trainable parameters; and $f$ is a nonlinear activation function. Through reasoning over the VisG, we can update node representation to capture more visual context information and to interact with SemG. Next we perform similar graph convolution [23] on the SemG for adjusting the representation of SemG, according to:

where $\widetilde{S}$ indicates an updated the graph representation of SemG; $A_{s}\in\mathbb{R}^{M\times M}$ is a learnable adjacency matrix or co-occurrence matrix that represents connections between semantic correlation or label dependency; $W_{s}\in\mathbb{R}^{D\times\ D}$ are trainable parameters. By performing a propagation of feature information from neighboring nodes, we can improve the representation of each semantic node.

In S2V step: we utilize the evolved SemG to promote contextual reasoning over the VisG $\widetilde{P}$. Specifically, to explore the relationship between two nodes from VisG and SemG, we compute their feature similarity as a guidance matrix $G^{s2v}\in\mathbb{R}^{N\times M}$. For one node $\widetilde{p_{i}}\in\mathbb{R}^{D}$ in the VisG $\widetilde{P}$ and one node $\widetilde{s_{j}}\in\mathbb{R}^{D}$ in the SemG, we can compute the guide information $G^{s2v}_{i,j}$ that represents the assignment weight of the node $\widetilde{s_{j}}$ in SemG to the node $\widetilde{p_{i}}$ in VisG as follows:

where $i\in\{1,...,N\},j\in\{1,...,M\}$ and $W_{p}\in\mathbb{R}^{D/2\times D}$ and $W_{s}\in\mathbb{R}^{D/2\times D}$ are a learnable matrix to further reduce the feature dimension. After obtaining the guidance matrix $G^{s2v}$, we can distill information from SemG to enhance the representation of VisG, according to :

where $W_{s2v}\in\mathbb{R}^{D\times D}$ is a trainable weight matrix, $\beta_{s2v}\in\mathbb{R}^{N}$ is a learnable vector with zero initialization and can be updated by a standard gradient decent. We use a simple sum to melt information from graphs, which may be alternatively replaced by other commutative operators such as mean, max, or concatenate. With the help of the guidance matrix $G_{s2v}$, we effectively constructed the correlation between visual regions and semantic concepts, and incorporate corresponding semantic features into the visual node representation.

In V2S step: we adopt a similar method elaborated in Equation(5) to obtain the guidance matrix $G_{v2s}\in\mathbb{R}^{M\times N}$. Formally, the guide information $G^{v2s}_{i,j}$ that can be calculated as follows:

where $i\in\{1,...,M\},j\in\{1,...,N\}$. After getting the guidance matrix $G^{v2s}$, we update the graph representation of the SemG for generating the exemplar-based SemG according to:

where $W_{v2s}\in\mathbb{R}^{D\times D}$ is a trainable weight matrix, $\beta_{v2s}\in\mathbb{R}^{M}$ is a learnable vector and initialized by zeros. We extract the exemplar-based semantic graph from VisG with the guidance matrix $G_{v2s}$. By combining the S2V and V2S steps, the proposed GI unit enables the whole model to learn more discriminative features for performing fine pixel-wise classification and generate a semantic graph for each input image.

Unit outputs:  The GI unit has two outputs, one is the exemplar-based SemG, which are described in detail in section 3.3, and the other is the VisG enhanced by semantic information. The evolved node representation of VisG can be used to enhance the discriminative ability of each pixel feature further. As previous methods [26, 25], we reuse projection matrix $Z$ to reverse project the $P_{o}$ to 2D pixel features. Formally, Given node features $P_{o}\in\mathbb{R}^{N\times D}$ of the VisG, the reverse projection (or Graph Re-Projection) can be formulated as follows:

where $W_{o}\in\mathbb{R}^{D\times C}$ is a trainable weight matrix that transform the node representation from $\mathbb{R}^{D}$ to $\mathbb{R}^{C}$, $Z^{T}\in\mathbb{R}^{L\times N}$ means the transposed matrix of $Z$, and we employ a residual connection [20] to promote the gradient propagation during training.

We propose a Semantic Context Loss or simply SC-loss to constrain the generation of exemplar-based SemG. It emphasizes the categories that appear in the scene and suppresses those do not appear in the scene, which makes the GINet a capable of enhancing the external semantic knowledge adaptively to each sample. Specifically, we first define a learnable semantic centroid $c_{i}\in\mathbb{R}^{D}$ for each category. Then, for each semantic node $\widetilde{s_{i}}\in\mathbb{R}^{D}$ in a SemG $\widetilde{S_{o}}$, we compute a score $v_{i}$ by performing a simple dot product with a sigmoid activation upon $\widetilde{s_{i}}$ and $c_{i}$. The $v_{i}$ ranges from 0 to 1 and is trained with the BCE loss. The SC-loss minimizes the similarity between the node feature in the semantic graph and the semantic centroid of nonexistent categories, and maximizes the similarity to existent classes. If $v_{i}$ is closer to 1, the corresponding category exists in the current sample; Otherwise, it does not exist. The SC-loss can be formulated as follows:

where $y_{i}\in\{0,1\}$ represents the presence of each category in ground truth. The proposed SC-loss is different from SE-loss in EncNet [54]. It built an additional fully connected layer on top of the Encoding Layer [54] to make individual predictions and was employed to improve the parsing of small objects. However, our SC-loss is employed to improve the generation of exemplar-based SemG.

We also add a full convolution segmentation head attached to Res4 of the backbone to obtain the segmentation result. Therefore, the objective of the GINet consists of a SC-loss, an auxiliary loss, and a cross-entropy loss, which can be formulated as:

where $\lambda$ and $\alpha$ are hyper-parameters, the selection of $\lambda$ is discussed in the experiment section, and $\alpha$ for auxiliary loss is set to 0.4 similar some previous methods [54, 55, 15, 46].

Pascal-Context [33] is a classic set of annotations for PASCAL VOC2010, which has 10,103 images. In the training set, there are 4,998 images. The remaining 5,105 images form the validation set. Following previous works [54, 55, 59], we use the same 59 most frequent categories along with one background category(60 in total) in our experiments.

COCO Stuff [4] has a total number of 10,000 images with 183 classes including an ’unlabeled’ class, where 9,000 images are used for training while 1,000 images for validation. We follow the same settings as in [15, 24], the results are reported on the data contains 171 categories (80 objects and 91 stuff) annotated for each pixel.

ADE20K [59] is a large scale dataset for scene parsing with 25,000 images and 151 categories. The dataset is split into the training set, validation set and test set with 20,000, 2,000, and 3,000 images, respectively. Following the standard benchmark [59], we validate our method on 150 categories, where the background class is not included.

During training, we use ResNet-101 [20] (pre-trained on ImageNet) as our backbone. For retaining the resolution of the feature map, we use the Joint Pyramid Upsampling Module [46] instead of the dilation convolution for saving the training time, resulting in stride-8 models. We empirically set the number of nodes in VisG as 64, and node dimensions as 256. Similar to prior works [8, 54], we employ a poly learning rate policy [7] where the initial learning rate is updated by $lr=base\_lr*(1-\frac{iter}{total\_iter})^{0.9}$ after each iteration. The SGD [39] optimizer is applied with 0.9 momentum and 1e-4 weight decay. The input size for all datasets is set to 520 $\times$ 520. For data augmentation, we apply random flip, random crop, and random scale (0.5 to 2) using the zero-padding if needed. The batch size is set to 16 for all datasets. We set the initial learning rate to 0.005 on ADE20K dataset and 0.001 for others. The networks are trained for 30k, 150k, 100k iterations on Pascal-Context [33], ADE20K [59], COCO Stuff dataset [4], respectively.

During the validation phase, we follow [54, 55, 46] to average the multi-scale {0.5, 0.75, 1.0, 1.25, 1.5, 1.75} predictions of network. The performance is measured by the standard mean intersection of union (mIoU) in all experiments.

We first conduct experiments on the Pascal-Context dataset with different settings and compare our GINet with other popular context modeling methods. Then we show and analyze the visualization results of our GINet. Finally, we compare with state-of-the-art to validate the efficiency and effectiveness of our method.

To further demonstrate the generalization of our GINet, we also conduct experiments on the COCO Stuff dataset [4]. Comparisons with state-of-the-art methods are shown in Table 3. Remarkably, the proposed model achieves 40.6% in terms of mIoU, which outperforms the best methods by a large margin. Among the current state-of-the-art methods, ACNet [16] introduced a data-driven gating mechanism to capture global context and local context according to pixel-aware context demand, DANet [15] deployed the self-attention module to capture long-range contextual information, and EMANet [24] proposesd the EMA Unit to formulate the attention mechanism into an expectation-maximization manner. In contrast to these methods, our GINet considers the operation of capturing long-range dependencies as the way of graph reasoning, and additionally introduces the semantic context to enhance the discriminant property for the features.

Finally, we compare our method and conduct experiments on the ADE20K dataset [59]. Table 3 compares the GINet performance against state-of-the-art methods. Our GINet outperforms the prior works and sets the new state-of-the-art mIoU to 45.54%. It is noting that our result is obtained by a regular training strategy in contrast to ANN [61], CCNet [21] and ACNet [16] where OHEM[38] is applied to help cope with difficult training cases. The ANN proposes an asymmetric fusion of non-local blocks to explore the long-range spatial relevance among features of different levels. The CCNet used a recurrent criss-cross attention module that aggregates contextual information from all pixels. We emphasize that achieving such an improvement on the ADE20K dataset is hard due to the complexity of this dataset.