Our pipeline is shown in Fig. 2. We put images into the Segmentation Network (S-Net) to get the predicted masks. Then we concatenate the images and predicted masks and deliver them into our rectification network to get the globally and locally rectified masks. We use the rectified masks to retrain the S-Net. The Rectification Network (R-Net) consists of three parts, that are the backbone, the Global Structure Module (GSM) and the Local Consistency Module (LCM). The GSM module is introduced in 3.2 and 3.3 including graph construction and hierarchical graph reasoning. The LCM module is introduced in 3.4. We introduce the integration method of the GSM and LCM in 3.5, and our semi-supervised training strategy in 3.6.

The global error is the predicted error of the entire human parts, such as confusing left and right arms. The local error means that pixels belonging to one human part may be assigned to two or more categories (e.g. the pixels belonging to dress are assigned to dress and upper-clothes). The GSM and LCM are different in the way the graph is built for different optimization objectives.

To parse the hard samples such as complicated postures or limbs absence, we propose a global graph model to explicitly represent the human-body features, and then optimize the model through an implicit graph model hierarchical reasoning. As shown in Fig. 2, the S-Net generates features of noisy masks ${\bf{F}}^{head}\in\mathbb{R}^{c\times w\times h}$, where $c$, $w$ and $h$ represent number of categories, the width and the height of the feature map, respectively.

To semantically model the features, we perform the one-to-one correspondence between the channels of the feature map and the nodes of the graph model. We globally transform the feature ${\bf F}^{head}$ into a semantically low-level undirected graph model ${\bf U}_{g}^{(low)}=\{({\bf X}_{g}^{(low)},{\bf A}_{g}^{(low)})\}$, where ${\bf X}_{g}^{(low)}\in\mathbb{R}^{c\times d}$ denotes the eigenvectors of graph nodes, $d$ denotes the dimension of each eigenvector, ${\bf A}_{g}^{(low)}\in\mathbb{R}^{c\times c}$ denotes the adjacency matrix of the graph model and $c$ equals to the category number of the dataset. ${\bf A}_{g(i,j)}^{(low)}$ denotes the spatial adjacency between the class $i$ and the class $j$. For the $i$-th channel of ${\bf F}^{head}$ corresponding to the $i$-th category, i.e., ${\bf F}_{i}^{head}\in\mathbb{R}^{w\times h}$, we transform it to a graph node as Eq. (1):

where $\varphi_{g}({\bf F})$ means reshaping the matrix ${\bf F}$ from $\mathbb{R}^{w\times h}$ to $\mathbb{R}^{1\times(wh)}$, ${\bf\Omega}_{i}\in\mathbb{R}^{(wh)\times d}$ is the projection matrix of the $i$-th category, ${{\bf X}_{g}}_{i}^{(low)}$ is the eigenvector of the $i$-th node, and $\times$ denotes the matrix multiplication. ${\bf\Omega}_{i}$ is a learnable parameter and can be trained end-to-end. The low-level graph model ${\bf U}_{g}^{(low)}$ contains the representation of human-body parts and the relations among them.

Because the human body structure is hierarchical [35], e.g., the upper body includes the head, the upper torso and the arms, the lower body includes the legs and the feet, the clothes include the coat, the pants and the dress, etc. We aggregate the low-level information of the relational graph nodes to perform higher-level structural information reasoning. In detail, we build the GSM to depict human structure information hierarchically and avoid the semantic structural errors by graph reasoning [16].

For the low-level graph model ${\bf U}_{g}^{(low)}$, we use Eq. (2) to perform graph reasoning for information propagation of nodes.

where ${\bf Z}_{g}^{(low)}$ is the node features after low-level graph reasoning, ${\bf W}^{(low)}$ is the trainable weight matrix and $n_{low}$ denotes the number of nodes in the low-level graph model.

We propose a trainable aggregation matrix ${\bf C}^{(agg)}$ to convert the low-level graph model to the high-level graph model. Then we obtain the high-level graph model ${\bf U}^{(high)}=\{({\bf X}_{g}^{(high)},{\bf A}_{g}^{(high)})\}$ by the Eq. (3) and (4),

where ${\bf X}^{(high)}$ and $n_{high}$ denote the feature of nodes and the number of nodes in the high-level graph model, respectively. For the high-level graph model, similar to Eq. (2), we perform message passing between nodes with Eq. (5) to get a new graph representation with more discriminative power,

Specifically, we compute the aggregation matrix with Eq. (6),

where ${\bf V}^{(low)}\in\mathbb{R}^{d\times n_{high}}$ is the trainable weight matrix, and $n_{high}$ is pre-set to control the number of nodes in the high-level graph model. The $n_{low}$ is equal to the number of categories $c$ here. This strategy makes our aggregation matrix ${\bf C}_{g}^{(agg)}$ trainable and improves the aggregating ability from low-level categories to high-level categories.

After aggregating the low-level categories (e.g., head, arm, pants and so on) into the implicit high-level categories (e.g., torso and clothes), we next revert the aggregated high-level model back to the low-level model. However, we cannot separate them by simply converting according to corresponding relations because they are one-to-many correspondence. Thus we perform decoupling processing to revert the aggregated information to the body-part level. After decoupling, a set of low-level graph nodes with stronger discriminative features is generated. Similar to the process of aggregation, we compute the trainable decoupling matrix by Eq. (7), and decouple the high-level graph model by Eq. (8) and (9) to generate the reverted eigenvectors of the low-level graph $\hat{\bf Z}_{g}^{(low)}$ and the adjacency matrix $\hat{\bf A}_{g}^{(low)}$:

In this way, we obtain the reverted low-level graph model $\hat{\bf U}^{(low)}=\{(\hat{\bf Z}_{g}^{(low)},\hat{\bf A}_{g}^{(low)})\}$ containing the high-level body structure information.

We apply a skip connection to integrate the original low-level graph with the reverted low-level graph,

We enhance the discrimination of the features by the trainable aggregating and decoupling strategy to correct the common global structural errors of human parsing, e.g., the confusion caused by the similar appearance of human parts. The eigenvector ${\bf Z}_{g}\in\mathbb{R}^{c\times d}$ of the reasoned graph model contains strong representation ability.

To apply hierarchical graph model to original feature maps, we convert the graph nodes ${\bf Z}_{g}$ to channel weights of ${\bf F}^{head}$ through Eq. (11):

where $GAP$ denotes the global average pooling operation and ${\bf a}\odot{\bf B}$ denotes the element-wise product between a vector ${\bf a}$ and a tensor $\mathbf{B}$, where

This strategy can increase the weights of categories with strong correlations and reduce the weights of the irrelevant categories.

In human parsing, another type of semantic error is the local consistency error. It means that pixels belonging to one human part may be assigned to two or more categories. This problem can be solved by computing the relation between pixels in an enlarged receptive field [2, 47]. The non-local algorithm [47] was proposed to calculate the spatial-wise correlation of all the pixels to obtain global semantic information, but the algorithm brings a large amount of computational redundancy. Similarly, it is not practical to set each pixel as a graph node and perform the graph reasoning directly, because the computational cost is too high to calculate the nodes relations.

Thus we project pixel features to graph nodes in our LCM. After projection, we indirectly obtain the relations between pixels by calculating the relations between nodes. By avoiding the local error, the performance can be improved through the enlarged receptive field within the controllable amount of calculation.

We perform projection from the input feature map ${\bf F}\in\mathbb{R}^{c^{\prime}\times w^{\prime}\times h^{\prime}}$ to a local graph model ${\bf U}_{l}=\{({\bf X}_{l},{\bf A}_{l})\}$ by Eq. (13). ${\bf X}_{l}\in\mathbb{R}^{(c\times d)}$ represents the node features. $c^{\prime}$ is the channel number of ${\bf F}$. It may not equal to the number of categories $c$.

where ${{\bf\Omega}_{l}}_{1}\in\mathbb{R}^{c\times c^{\prime}}$ and ${{\bf\Omega}_{l}}_{2}\in\mathbb{R}^{(w^{\prime}h^{\prime})\times d}$ represent the projection matrices, and $\varphi_{l}({\bf F})$ represents reshaping the matrix ${\bf F}$ from $\mathbb{R}^{c^{\prime}\times w^{\prime}\times h^{\prime}}$ to $\mathbb{R}^{c^{\prime}\times(w^{\prime}h^{\prime})}$. Then we perform graph reasoning similar to Eq. (2) to obtain the graph model ${\bf Z}_{l}\in\mathbb{R}^{c\times d}$. Note that the way to build the graph model in LCM is different from the way to GSM in 3.2.

We convert the graph feature ${\bf Z}_{l}$ to the weight of the feature maps, and perform the element-wise product of the weight and the input feature map ${\bf F}$ to get the spatially and locally rectified feature map ${\bf F}_{rectified}$ by Eq. (14),

By local graph reasoning, we capture the contexts and get the spatial relation of pixels indirectly. Because of the structural nature of human parts, we utilize the global information ${\bf Z}_{g}$ to adjust the local information ${\bf Z}_{l}$ for better local category consistency, and the first equation in Eq. (14) is transferred to

where $\alpha$ is the weight of global auxiliary.

We correct the global structural error and the local consistency error of the predicted masks by our GSM and LCM, respectively. The graph features of the two modules have different level of feature representation. Thus, we can not simply integrate them by addition. For the two types of errors, the local error is lower-level than the global one and should be corrected before the global structure error. It is worth noting that the process is not reversible. Thus the GSM and LCM are integrated by a cascaded way, as shown in Eq. (16).

where $\phi(.)$ represents multi-layer convolutions. Then we integrate these two rectification modules to obtain stronger representation ability.

Pseudo-labels for retraining may exploit abundant features of the object, making the segmentation network have stronger inference power. However, lots of noisy pseudo-labels are introduced and the network is not capable to rectify itself autonomously, causing error accumulation and network degradation. Simply using the originally predicted masks to retrain the segmentation network cannot enhance the performance effectively.

On the contrary, our rectification network is proposed to learn the error distribution of noisy predicted masks and reduce the global and local errors. The training process is elaborately shown as Alg. 1.

The baseline of our segmentation network is the CE2P algorithm [37], and the backbone of our rectification network is ResNet-101 [19], since our graph reasoning is performed at a high semantic level. The size of input images is $384\times 384$. We adopt data augmentation in training and retraining, like random scales (0.5 to 1.5), cropping and horizontal flipping. We both train and retrain our networks for 150 epochs in experiments under the semi-supervised setting. In the optimization process, we adopt SGD optimizer with momentum of 0.9 and weight decay of 5e-4. We use the “poly” learning rate strategy with an initial rate of 0.007. The node number of low-level graph model $c$ is a hyper-parameter, and $c$={20, 18} for {LIP, ATR} dataset. We set the batch size as 10 per-GPU.

We conduct elaborate ablation experiments for our rectification strategy and the graph reasoning modules. In section 4.3.1, we conduct experiments under semi-supervised settings to evaluate our retraining and rectification strategy. We adopt different amount of labeled and unlabeled samples for semi-supervised learning. In section 4.3.2, we perform more experiments under supervised setting to test our GSM and LCM.

We compare the proposed methods with several strong baselines on the LIP dataset, as demonstrated in Tab. 4. In the experiment setting, we treat the training set as labeled samples, and use the validation set and the test set as unlabeled samples. According to our proposed framework, we generate pseudo-labels for unlabeled samples, then use the labeled and augmented samples to retrain the segmentation network. As shown in Tab. 4, our algorithm outperforms the other state-of-the-art algorithms. Specifically, our method yields a mIoU of $56.34\%$, improved by $0.44\%$ compared to the HRNet [42] and by $3.2\%$ compared to the baseline CE2P [37]. Additionally, our model also outperforms the MuLA [32] ($49.3\%$), the MMANs [29] ($46.81\%$), the Attention [5] ($42.9\%$), and the Deeplabv2 [3] ($41.6\%$), respectively. MMANs [29] adopted adversarial network with unstable and intricate training process, and MuLA [32] proposed to jointly train the human parsing and pose estimation networks with tremendous cost of computation and time. But they only get limited performance in terms of mIoU. This confirms the effectiveness of our rectification strategy. Using unlabeled samples still makes sense for improving the representation and generalization power.

We add our rectification network on the top of the segmentation network, and retrain the latter one using the labeled data and the generated pseudo-labeled samples. The visual explanation of the rectification strategy is shown in Fig. 3. In the second row of the figure, the wrongly predicted areas are highlighted in the dotted rectangles. The CE2P algorithm correctly predicts the edges but fails to label the hard samples of semantic errors. The third row illustrates the attention maps for rectification. These maps are generated by calculating the gradient of the rectification network and corresponding to the original image. It is obvious that the rectified network focuses on the mistakenly predicted human parts. For example, for the hard samples of the back view (columns $2,3,4$), the samples without head (columns $7,8,11,13,14$), and the samples with complicated gestures (columns $10,12$), the rectification network can capture the errors and perform rectification as shown in the fourth row. These predicted errors in baseline are corrected effectively, especially for the confusing left and right limbs. Moreover, due to the high cost of accurate pixel-wise annotation, the ground truth of the LIP dataset is low-quality in detail. Our rectification mask performs even better than the ground truth in these details (column 8).