Automatic Image Labelling at Pixel Level

In the task of semantic segmentation, numerous method [3, 25] usually combine low-level and high-level features to boost performance. Compared to simply fusing low-level features with high-level ones, Zhang et al. [28] pointed out that later fusion is more effective by introducing high-resolution details into high-level features and embedding semantic information into low-level features.

Inspired by the works in [28, 29], we establish a Boundary Guided (BG) module that can recover the localization details of objects by introducing low-level features into high-level ones and embedding semantic information into low-level features. The structure of BG module is shown in Figure 1, where a boundary extractor is first applied to obtain the semantic features of object boundaries. Based on the observation that the pixel values in the non-boundary areas change slowly while the pixel values around the boundary change significantly, the boundary extractor can be designed as:

where $X$ represents the feature maps, $Maxpool(.)$ and $Minpool(.)$ are max-pooling and min-pooling operations respectively. In practice Eq. (1) can be implemented by $\mathcal{D}(X)=Maxpool(X)+Maxpool(-X)$. Thus, the boundary extractor makes that the values of pixels not around the non-boundary are close to zero while others are not (see Figure 2). After that, an element-wise multiplication is performed between the low-level features and the semantic features of object boundaries.

DCNNs have demonstrated their outstanding capability on the task of semantic image segmentation, but they have difficulty in obtaining good results with detailed pixel-level structures/boundaries. One core issue is that obtaining object-centric decisions from a classifier requires the invariance to spatial transformations, which inherently limits the spatial accuracy of the DCNNs [2]. Several methods have recently been proposed to alleviate this issue and they can be grouped into two main strategies. The first strategy is to build deep encoder-decoder networks which can preserve accurate object boundaries [3, 25]. The second strategy is to use CRF to refine the segmentation results iteratively [2, 30]. Although these approaches have achieved substantial improvements on semantic image segmentation, using multiple down-sampling layers may lead to more information loss.

The guided filter [31] is an edge-preserving operator and can transfer the structure of the guidance image (e.g., original RGB image or low-level features) to the filtering output. In this paper, we use the guided filter to extract the edge contour coefficient for each pixel from the guidance image, then such coefficients are used to weight the coarse object masks for generating the refined ones. More specifically, suppose that two inputs, the coarse object mask (e.g., the output of BG module) $p$ and the guidance image $I$, are fed to the guided filtering. The outputs of the guided filter (i.e., the refined masks) are obtained as follows:

where $i$ is the index of a pixel in $I$, $j$ is the index of a point in $p$, and $k$ is the index of a local square window $w$ with radius $r$. $W_{ij}$ is the edge contour coefficient of the pixel $(i,j)$ in the image $I$ and it is calculated as:

where $\left|w\right|$ is the number of pixels in the local square window $w_{k}$, $\mu_{k}$ and $\sigma^{2}$ are the mean and variance of the image patch in $w_{k}$, and $\varepsilon$ is a regularization parameter. In this way, we seamlessly integrate the recognition capacity of DCNNs and the fine-grained localization ability of guided filter. The object masks can further be refined by using the guidance image to fine-tune the output of DCNNs.

In the GF module, the low-level features are often used as the guidance image because they contain more object structures. In fact, the low-level features are too noisy to provide a high-quality guidance. Using the low-level features directly as the guidance image could be less effective for semantic image segmentation. Actually, the high-level features can guide the low-level features on recovering the localization details of objects. To this end, we propose High-level Guided Low-level (HGL) module, which integrates the high-level features to extract more discriminative low-level features:

where $HF(X)$ and $LF(X)$ are high-level features and low-level features, respectively. $\circledast$ represents the element-wise multiplication. The detailed structure of HGL module is shown in Figure 1. By using the HGL module, the quality of the guidance image can be improved, which can further boost the performance of semantic image segmentation.

The ResNet-101 model [32] has achieved promising performance on image classification and the Modified Aligned ResNet-101 used in DeepLabv3+ [3] has also demonstrated its strong potential for the task of semantic image segmentation. On top of that, we use Atrous Spatial Pyramid Pooling module (ASPP) proposed in the DeepLabv2 [2] to capture multi-scale information. With our designed BG, GF and HGL, we propose a Guided Filter Network (GFN) to achieve semantic image segmentation, as illustrated in Figure 1.

In this paper, we use the cross-entropy loss function for semantic image segmentation, which is formulated as follows:

where $C$ is the number of classes and $p_{ij}^{c}$ is the ground truth probability of the $c^{th}$ class at the pixel $(i,j)$. $S_{ij}^{c}$ represents the posterior probability of pixel $(i,j)$ belonging to the $c^{th}$ class. $h$ and $w$ are the height and the width of the image, respectively.

Compared with supervised learning methods that require manual labelling, the proposed approach requires initial object masks which can be automatically generated by transferring the segmentation knowledge learned from the source domain to the target domain. The learning of segmentation knowledge depends on the design of the pretext task, and a good pretext task can effectively improve the quality of the initial object masks. Considering that the goal of self-supervised learning is to generate initialization parameters for downstream tasks by proxy task that do not require human annotations, we design three pretext tasks to learn segmentation knowledge. After the pretext task training finished, the learned parameters are transferred to GFN to generate initial object masks in the target domain.

For a given set of object classes of interest, after the GFN is learned from the training images in the source domain (pretext task), it is then used to obtain the initial masks for the object classes of interest in the target domain. Because of the significant difference between the source domain and the target domain, the GFN may not be able to obtain the initial object masks accurately in the target domain. One reasonable solution is to fine-tune the GFN iteratively in the target domain. For this purpose, we treat the refined object masks as a pseudo label to optimize/refine GFN iteratively. Note that noisy labellings may lead to a drift in semantic image segmentation when they are used to supervise the training of GFN directly. Thus we define the reliability of a labelling (generated by our GFN) based on the area ratio, e.g., the proportion of the area between the automatically-labelled region and the whole image. The area ratio can be formulated as:

where $S_{annotation}$ represents the sum of pixels of the automatically-labelled region, and $S_{image}$ is the sum of pixels of the whole image. Only a subset of the pixels in the automatically-labelled region within a ratio between 0.1 and 0.9 are selected for network training, and this parameter can appropriately be adjusted according to the ratio of the object’s area in the image. We choose such parameters because the objects in our image set are neither too big nor too small. By continuously leveraging such training images with automatically-generated pixel-level labellings to optimize/refine our GFN in the target domain, the generated object masks may become more accurate gradually. After several iterations, we can finally obtain the fine-grained object masks with detailed pixel-level structures/boundaries in the target domain.

To train the GFN, a mini-batch size of 8 images is used. The network parameters are initialized by the ResNet-101 model [32] which is pre-trained on ImageNet [36]. The initial learning rate is set as 0.007 and divided by 10 in every 5 epochs. The weight decay and the momentum are set to 0.0002 and 0.9, respectively. In addition, in the Guided Filter (GF) Module, the values of $r$ and $\varepsilon$ are first determined by grid search on the validation set. We then use the same parameters to train GFN.

• •

  PASCAL VOC 2012: The PASCAL VOC 2012 segmentation benchmark [19] contains 20 foreground object classes and one background class. It has 1,464, 1,449, and 1,456 images labelled at the pixel level for network training, validation, and testing, respectively. The dataset is augmented by the extra labellings provided by [37], resulting in 10,582 training images.

• •

  “Orchid” Plant Image Set: We have crawled large-scale plant images which include 252 plant species (object classes) in the “Orchid” family, where 149,500 plant images are used to train the base network and other 500 images for testing.

• •

  FGVC Aircraft: FGVC Aircraft dataset [38] contains 10,000 images for 100 fine-grained aircraft classes. We divide 10,000 images into 9,000 training images and 1,000 test images.

• •

  CUB-200-2011: The CUB-200-2011 dataset [39] contains 200 fine-grained bird species. It has 11,788 images, of which 11,288 images for training, 500 images for testing.

• •

  Stanford Cars: The Stanford Cars dataset [40] contains 16,185 images for 196 fine-grained classes. We divide 16,185 images into 15,685 training images and 500 test images.

Note that the test images of the last four datasets are manually labelled at the pixel level. We use our proposed approach to generate pixel-level labellings, where the initial object masks are generated by our transfer learning approach.

It is worth noting that our transfer learning approach can generate initial object masks directly, where PASCAL VOC 2012 is treated as the source domain and our “Orchid” plant, FGVC Aircraft, CUB-200-2011, and Stanford Cars image sets are treated as the target domain. The reason is that the categories of “potted plant”, “aeroplane”, “bird”, and “car” in the PASCAL VOC 2012 is semantically-similar with all 252 plant species in our “Orchid” plant image set, 100 fine-grained “aircraft” classes in the FGVC Aircraft image set, 200 fine-grained “bird” classes in the CUB-200-2011 image set, and 196 fine-grained “car” classes in the Stanford Cars image set, respectively. Thus the knowledge learned from the PASCAL VOC 2012 dataset is transferable to our target datasets.

Our Branch image set contains 2,134 images, including 500 complex images and 1,634 simple images. Images are further divided into the training set and the test set, which have 2,034 and 100 images respectively. The 100 test images are manually labelled at the pixel level as the ground truth.

For the “branch” class, its semantically-similar category cannot be identified from public datasets, we use our proposed approach to generate pixel-level labellings, where the initial object masks are generated by our simple-to-complex approach.

We have also evaluated the impacts of the number of simple images with pixel-level labellings on the performance of our proposed method. As illustrated in Table I, one can easily observe that the performance of our proposed method can be improved when more simple images are incorporated for network training.

For the PASCAL VOC 2012 dataset, we use our proposed approach to generate pixel-level labellings, where the initial object masks are generated by the CAMs approach [33].

Figure 5 provides some of our segmentation results on the PASCAL VOC 2012 training set. From these experimental results, we observe that the automatically-generated labellings have precise segmentation masks even some labellings are better than ground truth. We attribute this to our proposed Guided Filter Network (GFN) that can transfer the structure of the object class of interest into the segmentation result to reason refined masks. Meanwhile, this indicates that manually labelling large-scale training images at the pixel level is very difficult to provide consistent labelling quality.

We use mIoU to verify the effectiveness of our proposed method and calculate its mIoU after every iteration until the mIoU no longer grows. The model with the best mIoU is then selected to generate the final pixel-level labellings. For the iterative training method, we start from the segmentation results for the first iteration, in later iterations, the segmentation results for the previous iteration are employed to train the segmentation network for the next iteration.

Figure 6 shows the performance of the recursive refinement. The performance of all these networks improves as the training round increases at first, and then saturates after few training rounds. Some examples for each step of our proposed approach are shown in Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11.

We use mIoU as our main evaluation metric. Table II summarizes the evaluation results on the mIoU over “Orchid” Plant, CUB-200-2011, Stanford Cars, FGVC Aircraft, Branch, and PASCAL VOC 2012 image sets. One can easily observe that our proposed approach can achieve good performance on generating pixel-level labellings (e.g., fine-grained object masks) with acceptable mIoU. Figure 12 and Figure 13 describe several fine-grained object masks with detailed pixel-level structures/boundaries on “Orchid” Plant, FGVC Aircraft, CUB-200-2011, Stanford Cars, Branch, and PASCAL VOC image sets. From these experimental results, one can easily observe that our proposed approach can automatically generate fine-grained object masks, and the annotation quality is very close to that of manually-labelled ones.

There is an experiment that could be carried out to shed some light in terms of performance. We provide the comparison results when the manually-labelled images and the automatically-labelled images (whose annotations at the pixel level are automatically generated by our proposed method) are employed for network training. Thus, we have evaluated two approaches on their mIoU when: (a) manually-labelled images are used, where we choose 800 manually-labelled images in the FGVC Aircraft image set as the training images and 200 as the test images; (b) automatically-labelled images are used, where the pixel-level labellings for these 800 training images are automatically generated by our proposed method. We use our GFN as the baseline model, where the parameters are initialized by the ResNet-101 model which is pre-trained on ImageNet. Table III summarizes the evaluation results on mIoU, one can easily observe that our proposed approach (i.e., leveraging automatically-labelled images for network training) can achieve very competitive performance.

In this subsection, the effectiveness of our proposed GFN is firstly evaluated over the PASCAL VOC 2012 segmentation benchmark [19] against numerous state-of-the-art techniques for semantic image segmentation, and we adopt the mean intersection-over-union (mIoU) as the evaluation metric on segmentation accuracy.