Mask Guided Matting via Progressive Refinement Network

A majority of matting methods requires a trimap as additional input, which divides an image into foreground, background, and unknown regions. Traditional methods are often sampling-based or propagation-based. Sampling-based ones [11, 7, 15, 33, 38] estimate foreground/background color statistics through sampling pixels in the definite foreground/background regions to solve the alpha matte in the unknown region. The propagation-based methods [6, 20, 21, 22, 35, 16], also known as affinity-based methods, estimate alpha mattes by propagating the alpha value from foreground and background pixels to the unknown area.

Recently, deep learning approaches have been proved successful in many areas, including classification [17, 36, 25, 23], detection [14, 2, 3], and segmentation [5, 42]. It also have achieved great success in image matting. [41] created a matting dataset with annotated mattes composited to various background images, and trained a deep network on it. Later, [30] introduced a generative adversarial framework to improve the results. [37] proposed to combine the sampling-based method and deep learning. [29] introduced a new index-guided upsampling and unpooling operations to better keep details in the predictions. [18] proposed a two-encoder two-decoder architectures to simultaneous estimate foreground and alpha. [24] further boost the performance with a contextual attention module.

It is noticeable that there are also some trials [1, 34] to get rid of the trimap to predict alpha matte. [45] proposed a framework consisting of a segmentation network and a fusion network, where the input is only a single RGB image. Later, [27] introduced a trimap-free framework consisting of mask prediction network, quality unification network, and matting refinement network for human portrait matting. The trimap-free matting performance is further boosted with attention module [31]. However, these trimap-free methods still have some gap to trimap-based ones in terms of performance. Another direction is to use an alternative guidance to trimap. [32] introduced a framework taking background images along with other potential priors (e.g., segmentation mask, motion cue) as additional inputs. It shows great potential and can obtain a comparable performance to state-of-the-art trimap-based methods.

Many conventional matting methods [11, 21] proposed to predict both alpha matte and foreground color for extracting foreground objects. However, it is only very recently [18] incorporated the foreground prediction into the deep learning framework. Later, [32] also predicts foreground color to reduces artifacts for a better composition result. Nevertheless, these methods mainly add a foreground decoder and directly learn from color label in [41], which only provides limited training samples and, more seriously, the color labels can be inaccurate and noisy(see Fig. 3). [10] proposes to use [21] to obtain a smoother color label.

Our method differs from algorithms mentioned above in the following folds: 1) Our model works in a more general setting where only an easy-to-obtain coarse mask, no matter user-defined or model-predicted, is needed as guidance. It could handle different qualities and even various types of guidance as input. Thus it could be used as either trimap-based or trimap-free model depending on what guidance is available. Our model could also leverage a stronger guidance to achieve even finer details. 2) Our methods could also predict the foreground color. Unlike [18], where the foreground prediction is directly learned from the color label, we note that the limited training data and inaccurate human label result in undesired results especially in the boundary regions. Instead, we propose to use Random Alpha Blending to avoid the bias in label, which not only introduces more diverse training samples but also avoid the inaccurate color label locating in boundary regions.

We follow previous methods to evaluate the results by Sum of Absolute Differences (SAD), Mean Squared Error (MSE), Gradient (Grad) and Connectivity (Conn) errors using the official evaluation code [41].

We adopt ResNet34-UNet proposed in [24] with an Atrous Spatial Pyramid Pooling (ASPP) [5] as the backbone for both PRN and color prediction. The first convolution layer is adjusted to take a 4-channel input consisting of a RGB image along with an external guidance input. Moreover, an alpha prediction head (Conv-BN-ReLU-Conv) is attached to the features at output stride 4 and 8 respectively to obtain side outputs.

To fairly compare with previous deep image matting methods, we train our MG Matting model using the Composition-1k datasest [41] which contains 431 foreground objects and the corresponding ground-truth alpha mattes for training. The network is initialized with ImageNet [8] pre-trained weight. We use crop size 512, batch size of 40 in total on 4 GPUs, Adam optimizer with $\beta_{1}=0.5$ and $\beta_{2}=0.999$. The learning rate is initialized to $1\times 10^{-3}$. The training lasts for $100,000$ iterations with warm-up at the first $5,000$ iterations and cosine learning rate decay [28, 12]. We also apply a curriculum learning manner to help the PRN training. Particularly, for the first $5,000$ iterations, the predictions of output stride 4 and 1 will be guided by guidance mask generated from ground-truth alpha, and for the next $10,000$ iterations, the guidance will be evenly and randomly generated from self-prediction and ground-truth alpha. Afterwards, each alpha prediction should fully rely on its self-guidance. The foreground color prediction is trained under the exactly same settings except that the generated training samples are composited by random foreground and alpha matte. It is noticeable that with RAB, we can add foreground color supervision on the whole image instead of only foreground regions, which produces more smooth and stable results (see Fig. 4).

The test set consists of 50 unique objects which are composited with 20 background images chosen from Pascal VOC [9], thus providing 1000 test samples in total. We note that since these synthetic datasets use PASCAL VOC images as background which may contain other salient objects, saliency/segmentation models may not be applicable to obtain a reasonable coarse mask. To best fairly compare MG Matting with other trimap-based methods, we test our model under two settings: 1) TrimapFG: We adopt the confident foreground regions in a trimap as a coarse guidance mask for our network; 2) Trimap: We normalize trimap to $[0,1]$ with the unknown pixels being $0.5$ and use this soft mask as guidance. We follow the the evaluation setting in Composition-1k which only computes the evaluation on the unknown region.

We summarize the alpha results and foreground color results in Table 1 and Table 3 respectively. We note that although our model is not trained with trimap, it still shows great robustness and transferability on these unseen types of guidance. Our model surpasses previous state-of-the-art models by a large margin. It also performs consistently considering the gap between trimap and trimapFG. We also note that our foreground color prediction not only reduces the errors significantly, but also produces much smoother results (see Fig. 4), which is desired in complex real-world scenarios where alpha matte can be noisy.

Distinction-646 [31] is a recent synthetic matting benchmark dataset, which improves the diversity of Composition-1k. It contains 1000 test samples obtained in a similar manner as Composition-1k. However, this dataset is released without official trimaps or other types of guidance, making it difficult to compare with previously reported results. Therefore, we use this benchmark mainly as a testbed to show how our method can refine a matte produced by another method.

We test a few state-of-the-art trimap-based baselines trained on Composition-1k. We firstly generate trimaps from ground-truth alpha mattes by thresholding and unknown region is dilated by kernel size 20. Then, we use these trimap-based methods to generate the matting results. Finally, we use these predicted alpha mattes as the guidance to our MG Matting method, and produce refined mattes.

As shown in Table 2, using the MG Matting as a refinement method consistently improves the results of other state-of-the-art methods. We also show the results reported by [31] in Table 2 for reference.

To validate the design of PRN and the introduced guidance perturbation, we conduct ablations studies as summarized in Table 4. Trimap is used as guidance masks in these experiments. However, we do not assume that the guidance type is known, so we purposefully do not use it to post-process the prediction by replacing the known foreground and background region. Instead, we report the two scores calculated over the whole image and the unknown region respectively for a more comprehensive evaluation of the robustness of our method.

We report ablations of different variants in Table 4. Baseline refers to a pure backbone without any add-ons. Adding side outputs and deep supervision to baseline improves the performance on both whole image or unknown area. We also try to use two convolution layers to fuse different outputs. However, linearly fusing the side outputs may not lead to better results. In contrast, the proposed PRN can better coordinate the semantic refinement and low-level detail refinement at different levels, thus obtaining a consistent improvement. We also show that the CutMask perturbation can further improve both the performance and robustness.

We also validate the effectiveness of RAB. We calculate the MSE and SAD of foreground color ($\mathbf{F}$) over foreground regions (\ie$\alpha>0$). The baseline achieves $\mathrm{MSE}=0.00623$ and $\mathrm{SAD}=82.30$, while with RAB, the performance is boosted to $\mathrm{MSE}=0.00321$ and $\mathrm{SAD}=62.01$.

We use the Composition-1k training set to train the model. Considering the semantic gap between the two datasets, we remove the transparent objects from the training data using the data list of [32]. Following [18], we also apply re-JEPGing, gaussian blur, and gaussian noises to the input image to make the model better adapt to real-world noises which are rarely seen in the synthetic dataset. Since these augmentations can change the color of the composited training image, thus the original color label may not be applicable. Therefore, we remove the composition loss from the supervision. Other training settings remain the same as in Sec. 4.

For trimap-based baselines, we follow [32] to generate trimaps from segmentation [44] automatically by labeling each pixel with foreground class probability $>0.95$ as foreground, $<0.05$ as background, and the rest as unknown, the unknown region is further dilated by $k=20$ to ensure it will not miss the long hairs. For our model, we threshold the segmentation at prob $=0.5$ to a binary mask.

We compare the results with state-of-the-art trimap-based methods DIM [41], GCA [24], IndexNet [29], Context-Aware Matting [18], and trimap-free method Late Fusion Matting [45] which is trained on Composition-1k training set and an additional portrait dataset. The results of baselines are obtained through either the open-source inference demos or the provided pre-trained weights.

We summarize the results in Table 5 under two settings: Whole Image, where the errors are calculated across the whole image, which can measure the overall quality; Details, where the errors are calculated only in manual-labeled regions containing hair details or other soft areas.

Compared to other methods, our model achieves a superior performance, especially regarding to the detail part, which illustrates its ability to capture the boundary details. We also note that the trimap-free method LFM performs badly, which could be caused by the fact that their portrait training data is not diverse enough and thus limits the the generalizability of their model (see Fig. 5 for examples).

We compare our results with another trimap-free method BSHM [27]. We contacted the authors and obtained the test results on a 100 images subset of our portrait dataset. Since [27] can only deal with low-resolution images, we downsample images to longer-side $720$, and the metrics are also computed on this scale. [27] achieves MSE $0.0155$ and SAD $10.66$ for whole image and MSE $0.0910$ and SAD $7.60$ for detail regions, while our MG Matting obtains a superior performance with MSE $0.0095$ and SAD $8.01$ for whole image and MSE $0.0637$ and SAD $5.94$ for details.