Joint Object Contour Points and Semantics for Instance Segmentation

There are two common ways of achieving instance segmentation: detection first and segmentation first. The detection first method consists of using the object detector to obtain the bounding boxes coordinates preferentially and then segment the instances in the bounding boxes. As an outstanding representative work, He $et~{}al$. [1] proposed Mask R-CNN to perform the instance segmentation task by adding a mask prediction branch on the basis of object classification and bounding boxes regression task of Faster R-CNN [4]. Many subsequent methods were proposed based on Mask R-CNN [6, 7, 8, 9, 2]. For instance, Chen $et~{}al$. [2]proposed MaskLab that utilized semantic segmentation and direction prediction to implement mask prediction. Huang $et~{}al$. [6] proposed MS R-CNN(Mask Scoring R-CNN) that added a mask IoU (Intersection over Union) head by combining instance features and corresponding prediction masks in Mask R-CNN to enhance the consistency between mask quality and mask score. Liu $et~{}al$. [7] proposed PAN (Path Aggregation Network) which combined more low-level features through Bottom-up Path Augmentation, and obtained more accurate segmentation through AFP (Adaptive Feature Pooling) and fully-connected fusion. Recently, there are also many excellent detection first methods that focus on building real-time instance segmentation systems [10, 11, 12, 13, 14, 15]. Contrary to the previous method, the segmentation first method is to classify each pixel in the image preferentially and then group them into a single instance. For instance, Hariharan $et~{}al$.  [16] proposed SDS (Simultaneous detection and segmentation) to locate segmentation candidate regions. This method uses multi-scale combinatorial grouping  [17] and obtain bounding boxes simultaneously to achieve instance segmentation. The InstanceCut [18] method utilized the edge graph to divide the segmentation map into different objects. Analogously, the work aforementioned is to perform segmentation tasks by classifying the foreground and background, and our work is to get a better instance segmentation effect by paying attention to the boundary of the object.

MTL (Multi-Task Learning) is an inductive transfer method that makes full use of domain-specific information hidden in the training signals of multiple related tasks. During backward propagation, multi-task learning allows shared hidden layer-specific features to be used by other tasks. In particular, MTL can learn features that are applicable to several different tasks. Such features are often not easy to learn in a single-task learning network. MTL is widely used in computer vision [6, 8, 9], speech, natural language processing, and other fields [19]. In [20], Su $et~{}al$. calibrates missing features in the learning process by combining low-level raw binary attributes and intermediate attributes. In [21], Abdulnabi $et~{}al$. proposed the use of a combination of hidden matrix and multi-matrix to decouple model parameters to allow CNN (Convolutional Neural Network) models to share visual knowledge between different categories simultaneously. In [22], Yuan $et~{}al$. enhanced the robustness of model coefficient estimation by constructing a hierarchical sparse learning model of multi-task interconnection. In [23], Yim $et~{}al$. added an additional task to improve the ability of DNN (Deep Neural Network) to maintain face identity, which using a human face and a binary target pose code to generate a face image with the same identity and target pose. In our method, we use the keypoint detection task in conjunction with the instance segmentation task in the original Mask R-CNN to calibrate the boundary features of the object, the purpose of which is to enhance the gradient flow between different tasks and the fusion of different attribute features.

Keypoint detection technology has been widely used in computer vision research, such as human pose detection tasks [24, 25, 26] and anchor free object detection tasks [27, 28, 29, 30]. The purpose of human body keypoint estimation task is to detect $k$ 2D keypoints of the human body, such as joints, facial features, etc., and describes human bone information by these keypoints. Lately, keypoint estimation techniques are also used in many detection tasks. Generally, the current object detection method can be divided into anchor base [31, 32, 33, 34, 35, 4] and anchor free according to whether the proposal is obtained by sliding window. Specifically, the anchor base method utilizes sliding window to generate a bunch of candidate anchor boxes, and then remove the redundant bounding boxes by NMS (Non-Maximum Suppression) [36] to get the final detection results. The anchor free method utilizes keypoint detection technology [37, 38] to obtain the center point or extreme point of the object for target positioning. By contrast, the detection points of this method are all located in the interior of the detected object, so it is more conducive to the learning of network features. Following that intuition, we utilize the keypoint detection branch to detect the edge aggregation points of the target to capture more geometric features of objects.

Since most of the existing instance segmentation methods are implemented by pixel-by-pixel classification, these methods depend on finer label information relative to the object detection task. For this reason, making a instance segmentation dataset will be a major expenditure of time and effort. During the annotation process, the annotator needs to obtain the edge contour of the instance by connecting the edge points for building the instance’s mask, so the boundary point features of the instance can well represent its geometric information. We all know that deep learning is a typical data-driven method. Therefore, in order to improve the accuracy of instance segmentation when the amount of data is limited, inspired by the artificial annotating process, we obtain a fixed number of edge aggregation points by uniformly sampling the edge contours of the groundtruth mask, and then utilize the newly added keypoint prediction branch to predict these edge aggregation points. Finally, the binary segmentation mask is obtained by fusing the edge information prediction and the mask prediction.

Next, we briefly review the framework of Mask R-CNN. Mask R-CNN is extended on the basis of Faster R-CNN, in addition to the shared backbone network and FPN (Feature Pyramid Networks), it in parallel adds a new branch for predicting the object mask on the bounding box recognition branch. The network can also be easily extended to other tasks, such as person keypoint detection (i.e. Keypoint R-CNN). In Keypoint R-CNN, its keypoint head is very similar to the mask head of Mask R-CNN. The mask head obtains an output resolution of $28\times 28$ through four $3\times 3$ convolutional layers with a channel of 256 which followed by an up-sampling layer, and finally employ a convolutional layer to make its dimensions consistent with the object category. Meanwhile, the keypoint head obtains an output resolution of $56\times 56$ through eight $3\times 3$ convolutional layers with 512 channels and followed by two upsampling layers. Besides, in order to implement Human Pose Estimation in Mask R-CNN, it uses one-hot mask to encode each keypoint position, and the structural design is similar to the mask branch, which uses fully convolutional network to predict a binary mask for each keypoint.

To train our keypoint detection subtask, we need the labels of the object edge points. But the dataset used for instance segmentation only provides binary groundtruth mask, so we need to make labels for keypoint detection subtasks ourselves. In the keypoint detection task, a heat map is generated for each detection point. Thus, for different instance targets in the training data, we need to extract the contour boundary points of the object to form training labels. However, the CNN (Convolutional Neural Network) network require the input training data with a constant channel dimension, hence we need to extract a fixed number of boundary points correspondingly. In this work, we generate training labels for keypoint detection by sampling the instance contours, as shown in the Fig. 3. But due to the large difference in the scale of the instances in the training dataset, the number of points obtained by sampling may be small. In this regard, we complement it to the required amount by randomly selecting existed sampling points. Besides, we also explored the different sampling method, which will be explained in detail in the experimental section.

According to the definition in [46], we can use symbols $\{\tau_{i}\}_{i=1}^{m}$ to describe multi-task training deep learning models, where $m$ represents the number of subtasks. For our framework $m=4$, it includes the bounding box regression task, the classification task, the target segmentation task, and the newly added keypoint detection task. Suppose that our dataset $D$ contains a total of $n_{i}$ training samples, then $D=\{x_{i}^{j},y_{i}^{j}\}$, where $x_{i}^{j}$ is the $j-th$ training instance in $\tau_{i}$ and $y_{i}^{j}$ is its corresponding training label. In our approach, we used the same training data for different learning tasks with different training labels, i.e. $x_{i}=x_{l},y_{i}\not=y_{l},and(i,l)\in\mathbb{R}^{m}$. More formally, the training label for the regression task of the bounding box is the coordinate of the center point of the groundtruth and its width and height. And for the target segmentation task, the training label is the corresponding binary mask, whereas for the keypoint detection task, the training label is the edge aggregation point obtained by sampling the target boundary.

Fig. 1 shows the general architecture of our model. In our pipeline, the box head and mask head are standard components of original Mask R-CNN. Obviously, we can simultaneously perform keypoint joint training with other heads by adding the keypoint detection branch. Suppose that we utilize $k$ edge aggregation points and the center point of the object, a total of $k+1$ points, as the keypoint labels information for training. And then a $M\times M$ heat map will be generated for each keypoint in the keypoint detection task of Mask R-CNN, so we can get a $M\times M\times(k+1)$ output mask tensor eventually. As for the mask prediction branch, a $N\times N\times C$ output tensor is produced through the fully convolutional network, where $C$ represents the total number of object categories. In addition, for capturing the overall edge information of the target, we do the channel-wise addition on the output features of the keypoint prediction branch for mapping the information of multiple keypoint with one hot encoding onto a single heat map. And in the next steps we reduce the heat map into a feature map with the same spatial scale as the mask prediction branch output by applying a $3\times 3$ convolution layer. In the end, we broadcast the edge information of the target to each channel of the mask prediction branch through feature fusion operation. There is one more thing should notice that we do not use ReLU (Rectified Linear Unit) activation function after the $3\times 3$ downsampling convolutional layer. As a consequence, our model can pay more attention to the segmentation edge of the object through the direct prediction of the object boundary point and the indirect fusion with the object segmentation feature output by the mask head.

The above fusion process could be mathematically described as follows:

where $\otimes$ represents channel-wise multiplication, $s$ stands for the stride of the convolution operation, $f_{k+1}$ stands for the output features of the keypoint prediction branch, and $f_{m}$ stands for the output features of the mask prediction branch.

In original Mask R-CNN, the multi-task loss is defined as three parts: $L_{cls}$, $L_{box}$, and $L_{mask}$. For our model we calculate the loss in the keypoint detection task by using the cross entropy function, which is the same as defined in Mask R-CNN. After the keypoint detection is added to our model, the loss function can be updated as follows:

where $\alpha$ represents the weight parameter to balance the loss of the keypoint detection task.

We verify our algorithm on the Cityscapes dataset, which contains a variety of stereo video sequences recorded in street scenes from 50 different cities. It provides 5000 images with fine annotations of a fixed resolution of $2048\times 1024$, which are splitted into 2975, 500 and 1525 images for train/val/test, and the test dataset labels are not publicly released. We evaluate results based on AP (Average precision) and AP₅₀. We conducte plenty of ablation experiments on the validation set and provided results on the test dataset by uploading our results to the server. We compare our model with other state-of-the-art methods on Cityscapes test subset, and the results are shown Table  1. Training on “fine-only” data, our method outperforms Mask R-CNN by 2.7% on test subset. Moreover, in order to verify that our method is not limited to a specific dataset, we perform experiment on COCO dataset to verify the generalization of our method. And in COCO dataset, it provides 115k images for training and 5k images for verification, and 20k images for testing.

We perform our experiments based on the torchvision detection module with a pytorch backend [47]. We take 4 images in one image batch for training. The shorter edges of the images are randomly sampled from [800, 1024] for reducing overfitting. Each training is carried out on two nvidia titan xp GPUs. For instance segmentation task the mask spatial scale is $28\times 28$, and for keypoint detection task is $56\times 56$. We randomly horizontal flip the input image with a probability of 0.5 to augment the dataset. Because we use the different number of GPUs, different training strategies are used for Citysacpes dataset compared with Mask R-CNN. We train our model a total of 64 epochs, with a learning rate of 0.005 for the fore 48 epochs and 0.0005 for the remaining 16 epochs. Following Mask R-CNN, we use SGD for gradient optimization with weight decay 0.0001 and momentum 0.9. ResNet-50 FPN [48] is taken as the initial model on this dataset, if not specially noted. And for keypoint detection task we use uniform sampling with 100 sampling points. For the COCO dataset we train our model for 24 epochs, with an initial learning rate of 0.005, and reduce it by 0.1 after 16 and 22 epochs, respectively.

The structural design choices for feature fusion: In our framework, feature fusion between multiple tasks is required, but the mask spatial scale of instance segmentation task and keypoint detection task is different, $56\times 56$ vs $28\times 28$. Therefor, we need to ensure that the input features of the fusion operation have the same spatial scale. There are a few design choices shown in Fig. 4 and explained as follows:

• (a) The output of keypoint predictor is size of $56\times 56\times(k+1)$, and the output of mask predictor is size of $28\times 28\times C$. After we do channel-wise addition on the keypoint predictor output, the feature map spatial scale is reduced to $28\times 28$ by using a $3\times 3$ convolution layer with a stride of 2.

• (b) The output of keypoint predictor and mask predictor is the same as that of design (a). The difference is before we do channel-wise addition on the keypoint predictor output, the feature map spatial scale is reduced to $28\times 28$ by using a $3\times 3$ convolution layer with a stride of 2.

• (c) The output of keypoint predictor is size of $56\times 56\times(k+1)$, and the output of mask predictor is size of $28\times 28\times C$. we upsample the output spatial scale of the mask predictor to $56\times 56$ using transposed convolutional layer.

• (d) The output of keypoint predictor is size of $28\times 28\times(k+1)$, and the output of mask predictor is size of $28\times 28\times C$.

The experimental results are shown in Table 2. We can see that each methods can achieve performance improvement except method (a), because in method (a) we only use one convolution layer to process the prediction results with a single channel, which will lose a lot of original edge prediction information.

The spatial scale reduction designs for feature fusion: Based on the design of the edge fusion structure, we try to reduce the spatial scale of the keypoint prediction output by using three methods: max pooling, average pooling, and a convolution layer with kernel size of 3 and stride of 2.

Table 3 shows the results for above spatial scale reduction designs, we can see that all the three methods can improve the performance of the model, which further validate the effectiveness of our method. Notably, we can see that compared with a simple pooling operation, for our model, using a convolutional downsampling method with learnable parameters performs better.

The choices of the fusion mode: After channel-wise addition and spatial scale reduction of the keypoint branch output, we can perform the fusion operation. In our experiment, we test three methods of maximum, multiplication, and addition to choose the best fusion strategy. Results are shown in Table 4, we can see that the fusion method of multiplication can be better adapted to our model.

The model we designed requires training tasks including box head, mask head, and keypoint head simultaneously. But the Cityscapes dataset only provides instance groundtruths masks, so we need to get the bounding box and edge aggregation point labels of the object from the mask of the instance ourselves. For a single target, the bounding box label is uniquely fixed, but for the edge aggregation point label we need to sample it from the edge contour of the target, which means that we have more alternative strategies. Specifically, we separately analyze the impact of corner sampling and uniform sampling on our method, and the results are shown Table 6. Analogously, corner sampling reflects the key geometric information of the object, whereas uniform sampling reflects the overall geometric information of the object. From the experimental results we can see that it is more conducive to the instance segmentation by using keypoint detection technology to capture the overall geometric position information of the object. Additionally, for the case of the number of edge sampling points is less than the required number due to the target being too small when sampling the edge points, we randomly select the existing sampling points for padding.

Furthermore, we demonstrate the influence of different sampling points number on the model’s performance, the results are shown in Table 5. Specifically, we conduct experiments on 50 and 100 sampling points respectively. And we find in the course of the experiment that the further increase of the sampling points number will damage the effectiveness of the model, which can be attributed to we add too much random information in padding operation, since there are a large number of small objects in the dataset. Moreover, in our approach, we use not only the points sampled from the object contour but also the center point of the object when we perform the keypoint estimation task, which is follow the intuition that the geometric relationship between the object boundary point and the center point can be learned during the model training process for improving the estimation accuracy of the boundary point. For example, the center point of the object can be regarded as the origin in a polar coordinate system, for the objects with the same category, the distance and angle of the target boundary point relative to the center point have certain regularity. To verify our analysis, we evaluate the effect of adding the object center point using COCO evaluation metrics, as shown in the Table 7, where AP^bb represents the AP value of the bounding box detection results, AP^mask and AP^keypoint represent the AP value of the mask predict results and keypoint estimation results respectively, and the subscripts $s$, $m$, and $l$ represent small, medium, and large objects, respectively. We can see that our method can simultaneously improve the performance of instance segmentation, keypoint estimation and object detection tasks.

In our model, keypoint detection is only an auxiliary task, and we hope that it can complete its duties while not affecting other tasks as much as possible. Therefore, we balance the importance of each task in the multi-task joint training by setting the attenuation coefficient of its loss function. The experimental results are shown in the Table 8, from the results in the Table, we can see that our method is very sensitive to the set balance coefficient, and we can get the best performance when we set $\alpha$=0.5.

Compared with the Cityscapes dataset, the COCO dataset contains objects of more classes and scales, so it is more challenging for our algorithm. Note that we set the loss balance weight coefficient $\alpha$ to 0.1 experimentally on COCO dataset. The experimental results are shown in Table 9. Notably, compared with baseline, our method has effective gains in both instance segmentation and target detection tasks. Moreover, in Table 10 we compare our results with state-of-the-art counterparts on COCO test-dev dataset, and it can be seen from the experimental results that we have obtained comparable performance.

In order to analyze the performance of our algorithm more intuitively, we show some concrete examples of our method and Mask R-CNN baseline in Fig. 6. It can be seen from the figure that, compared with the baseline, our method can obtain a segmentation effect that fits the contour of the object better, which is thanks to our newly added keypoint detection and auxiliary task and effective fusion strategy.