Unsupervised Shadow Removal Using Target-Consistency Generative Adversarial Network

Natural scene shadow removal has been widely studied. Early methods usually focused on physically modeling the shadow image and estimating the model parameters in terms of gradient domain manipulation [9, 11, 2], shadow illumination and color transfer [51, 12, 10, 7], and accurate shadow matte [34, 42, 44]. Recently, driven by shadow removal datasets with paired shadow and shadow-free images [28, 19], deep learning-based methods are widely used to analyze the potential mapping relation between shadow images and corresponding non-shadow images [4, 37, 29, 20]. Qu et al.[28] proposes an end-to-end multi-context embedding network to integrate high-level semantic context for shadow removal. Zhang et al.[30] uses a unified end-to-end framework to explore the residual and illumination for shadow removal. For learning shadow detection and removal jointly, Wang et al.[19] proposes a stacked conditional generative adversarial network, and Hu et al.[47] presents a deep neural network for shadow detection and removal by analyzing the spatial image context in a direction-aware manner.

Since now, a few of methods have explored the unsupervised shadow removal on the unpaired dataset and the only unsupervised dataset has been proposed in [48]. In paticular, Hieu and Dimitris [26] use shadow and non-shadow patches cropped from the shadow images and train a physical shadow formulation-based model via an adversarial framework. Hu et al.[48] present a Mask-ShadowGAN that learns shadow removal from the unpaired training data. The method takes use of the cycle consistency of CycleGAN [21], and with the guidance of the generated binary mask in the generative process, it facilitates the bidirectional mapping to transform the shadow image into shadow-free image.

Unsupervised shadow removal can be regarded as a domain mapping problem. Recent domain mapping works have used adversarial learning [18] to solve unpaired domain mapping [23, 53, 50]. Generative adversarial networks [18] are powerful adversarial learning models, which learn a generator and a discriminator jointly such that the generator produces realistic images that confuse the discriminator [35, 49, 46]. The cycle consistency constraint based methods [21, 24, 54] learn a bijection mapping by enforcing cycle consistency between the reconstructed data and input one to produce convincing mappings from the source domain to the target domain without paired supervision. Since then, many subsequent methods have been proposed to improve the cycle-consistency constraint [3, 36, 22, 14, 6]. Furthermore, for one-sided unsupervised domain mapping, Benaim and Wolf [43] propose to maintain the distances between samples within domains, and Fu et al.[17] develop a geometry-consistent generative adversarial network to perform image-to-image translation.

For the input shadow image $I_{X}$, we design two generators $G_{1}$ and $G_{2}$ to learn two separate $G_{X2Y}$ mappings, deriving two shadow-free images $I_{y1}$ and $I_{y2}$ with respect to $I_{X}$, respectively. Here, on the one hand, in order to confine $I_{y1}$ and $I_{y2}$ to be content consistent, we propose a target consistency constraint to act on the ends of the two generators. On the other hand, the two generators are acquired to produce two different representations of $I_{X}$ to mimic the assumption that two different shadow-discrepant images correspond to the same non-shadow image.

The two generators are mainly composed of two encoder-decoders with dual network structure, which we term as shadow residual generators, with $G_{res1}$ and $G_{res2}$ indicating the related sub-network in Figure 2. The two sub-networks are expected to generate two different shadow residual images from $I_{X}$. In particular, the encoder in each of the generator is a shadow transform encoder (denoted by $G_{STE1}$ and $G_{STE_{2}}$ for each encoder in Figure 2). During network training, $G_{STEi},(i\in\{1,2\})$ learns to transform $I_{X}$ from image domain to a kind of feature compositions, i.e. $F_{i}=G_{STEi}(I_{X}),i\in\{1,2\}$, which encodes the inherent relations between the non-shadow and shadow residual features with respect to $I_{X}$. The subsequent shadow residual decoders $G_{SRD1}$ and $G_{SRD2}$ are designed to decode the feature representations from feature domain to shadow residual images, i.e. $G_{resi}(I_{X})=G_{SRDi}(F_{i}),\in\{1,2\})$, layer by layer. At last, by applying the element-wise addition between $I_{X}$ and $G_{res1}(I_{X})$ and $G_{res2}(I_{X})$, respectively, the two separate shadow-free images $I_{y1}$ and $I_{y2}$ can be recovered.

In order to ensure $G_{1}$ and $G_{3}$ producing two separate shadow-free images from $I_{X}$, we intentionally initialize the dual sub-networks with different parameters, and the differences are maintained during the whole training process through different adversarial optimization objectives. Ideally, $I_{y1}$ and $I_{y2}$ will be exactly the same images under the target consistent constraint, however, as they were trained by different adversarial generators, we build a model selection module (MSM) to make a selection from the two shadow-free images. The MSM module is a pre-trained shadow/non-shadow binary classifier, which is embedded at the end of the generative network in TC-GAN and only applied for the network inference phrase. Through comparison of the output probability of the two classes, it selects the one with higher confidence from the two generated images ($I_{y1}$ and $I_{y2}$) to be a real non-shadow image as the final output.

To summarize, the overall process of the generative network of TC-GAN can be formulated as:

In model implementation, we build the shadow transform encoders with three convolution layers (stride-2) and nine residual blocks [54] , and the shadow residual decoders are designed to have three stride-2 deconvolution layers. Each layer in both encoders and decoders is followed by an instance normalization and ReLU activation function. The structure of MSM is built by four convolution layer with stride of 2, and each layer uses the same nomalization and activation function with encoders.

Similar with the generative network, there are two discriminators $D_{1}$ and $D_{2}$ corresponding to the two generators and with the same network structure in the Discriminative network. The discriminator basically contains four stride-2 convolutional layers, each of which is followed by an instance normalization and Leaky ReLU activation function (slope of 0.2).

In the GAN based image-to-image translation approaches for unpaired data, image data in the same domain is believed to share some common characteristics. Delighted by the assumption and the fact that images from the non-shadow domain have agreement on some aspects like consistent illumination and brightness, etc., we facilitate each discriminator in the dual discriminative network for adversarial training by feeding a randomly selected sample from non-shadow image dataset as the real non-shadow image (as we denote by $I_{rn1}$ and $I_{rn2}$ for $D_{1}$ and $D_{2}$ in Figure 2) for discrimnative supervision. In addition, to leverage the two generators for individual learning, we intentionally choose two different real non-shadow images from the dataset, i.e., $I_{rn1}\neq I_{rn2}$, for the two discriminators.

As before mentioned, the target-consistency constraint is proposed to confine the correlation between $I_{y1}$ and $I_{y2}$. Specifically, given an input sample $x$ being drawn from the shadow image data distribution $P_{X}$, the output targets of the two generators $G_{1}$ and $G_{2}$ are restricted to be as close as possible by applying the target-consistence constraint:

The key role of target consistency constraint is to connect the dual sub-GAN networks ( $G_{1}$-$D_{1}$ and $G_{2}$-$D_{2}$) for the overall learning of TC-GAN, but more than that, it drives an adversarial learning between the consistency of $I_{y1}$ and $I_{y2}$ and distinctiveness of $G_{1}$-$D_{1}$ and $G_{2}$-$D_{2}$, in addition to each adversarial constraint of $G_{1}$-$D_{1}$ and $G_{2}$-$D_{2}$. Furthermore, the target-consistence loss restricts the output shadow-free targets directly, whereas random disturbance would be induced by using the binary mask guided cycle-consistence constraint in [48], leading to ambiguous textures in the recovered shadow-free image as shown in Figure 3 (c).

For training each sub-GAN network, we use the generic adversarial constraint [18] to enforce the generator and discriminator to jointly optimize the adversarial loss. Specifically, let $p(X)$ and $p(Y)$ represent the data distribution of shadow domain $D_{X}$ and non-shadow domain $D_{Y}$, respectively, the adversarial loss $\mathcal{L}_{gan}(G_{i},D_{i})$ for each sub-GAN $i\in{1,2}$ can be expressed as:

where $y_{1}$ and $y_{2}$ are the two different real non-shadow images $I_{rn1}$ and $I_{rn2}$ for supervising true shadow-free image restoration.

In addition, by following most of the GAN-based image-to-image translation works on unpaired data [52], we further use the identity loss to leverage the generators learning from real non-shadow images. For the purpose of (i) preserving the consistency between the generated shadow-free image and input image on the non-shadow areas in terms of color or other appearance features [21]. (ii) making faster convergence for training on both generators, the identity loss of $G_{1}$ and $G_{2}$ for non-shadow data $y_{1}$ and $y_{2}\in P(Y)$ are described as:

In summary, the final loss function for TC-GAN ($\mathcal{L}_{tcgan}$) is a weighted sum of the adversarial loss of the both sub-GANs, target-consistency loss, and identity loss:

where $\lambda_{1}$,$\lambda_{2}$ and $\lambda_{3}$ are trade-off hyperparameters to weight the contribution of $\mathcal{L}_{gan}$, $\mathcal{L}_{tc}$ and $\mathcal{L}_{idt}$, respectively. In this work, We empirically set $\lambda_{1}$,$\lambda_{2}$ and $\lambda_{3}$ as 1, 40, and 5 separately in all the experiments. Furthermore, the negative log likelihood objective is replaced by a least-square loss for adversarial loss, which is shown to be more stable and effective than the original log likelihood according to [49]. At last, we optimize the whole network in a minimax optimizer.

In training process, we use the Adam solver [38] with a basic learning rate of 0.0002, and assign the first and second momentum values as 0.5 and 0.999 for network optimization. The parameters for all the generators and discriminators are random initialized by a zero-mean Gaussian distribution with a standard deviation of 0.02. The network is trained in a total of 200 epochs with each mini-batch size of one. In addition, we fix the learning rate for the first 100 epochs, and gradually reduce it to be zero by applying a linear decay rate for the rest of epochs. All the images are first re-scale to 286 ×286 and randomly cropped to be size of 256 ×256 as inputs. Lastly, we built our model on PyTorch library and trained on a computer with a single NVIDIA GTX1080Ti GPU.

The supervised dataset: The ISTD [19] is a widely used dataset for supervised shadow removal. It contains 1330 training triplets and 540 testing triplets of shadow image, non-shadow image and binary shadow mask. The shadow images are acquired by artificially masking some shadowless regions of the non-shadow natural images. However, The artificial shadowed images, would only cover limit shadow patterns, and visible differences between the actual shadowless images and their non-shadow labels would be induced, as have been demonstrated in [48].

The unsupervised dataset: Up to now, the USR [48] dataset is the first and the unique unsupervised shadow removal dataset. It contains 2445 shadow images (1956 for training and 489 for testing) and 1770 non-shadow images. Note that the 1770 non-shadow images are not correlated with any image in the 2445 shadow images. Moreover, the USR dataset is captured on a thousands of different outdoor scenes and shaded by various types of objects, which covers the most number of different scenes and shadow patterns over other existing datasets.

Since the difficulty and feasible data has not been available for a long time, the topic of unsupervised shadow removal has rarely been investigated. Mask-ShadowGAN [48] is the state-of-the-art unpaired shadow removal methods and has shown to have good performance on the USR dataset. In addition to it, we compare our TC-GAN with some latest unsupervised image-to-image translation models, which are trained on the USR dataset by using the official provided source code and hyper-parameters. Specifically, the compared models are: GAN-alone (Pix2Pix [40] model without paired supervision), DistanceGAN [43], CycleGAN [21], U-GAT-IT [22] and AttentionGAN [14]. Noted that in order to make fair comparison, we generate shadow residual for all the generators as described in Section 3.1, and finally add it to the input shadow image to obtain the shadow-free output.

In order to further evaluate our methods, we compare our TC-GAN with some supervised methods on the supervised ISTD dataset as well. Here, many baseline and advanced shadow removal methods are take into comparison on the ISTD dataset, including Gong et al.[13], Guo et al.[42], Yang et al.[41], ST-CGAN et al.[19] and DSC [47]. Specially, we obtain their results and source code directly from the official website and in order to make a fair comparison, we re-scale the size of all the testing images to 256 × 256 via cubic spline interpolation. In addition, since TC-GAN is independent of shadow mask information, which is provided by the ISTD dataset, while many supervised have used it as the guidance to recover the shadow-free image, we do not compared with more state-of-the-art methods that use full ISTD for training. We evaluate the performance of all the supervised methods by calculating RMSE between the generated shadow-free image and the annotated ground truth by considering several validation areas: (i) in shadow area (S), (ii) non-shadow area (N) , (iii) the entire image (A) respectively. In addition, by considering that the non-shadow labels in the ISTD dataset contain noises, we further measure the RMSE of the generated shadow-free image and the input shadow image in the non-shadow area (N-I).

For our TC-GAN on the ISTD dataset, we disrupt the paired order of shadow images and non-shadow images during training, and use random cropping to avoid pixel matching between any two shadow images and non-shadow image in the dataset. We finally summarize all the evaluation results Table 2. As can be seen that our TC-GAN achieves comparable performance over the supervised methods, especially for the non-shadow area (N-I) results, and importantly, TC-GAN is trained without paired supervision.

Figure 5 illustrates the visual comparison of shadow removal results on ISTD dataset. It can be seen that due to the noise of training labels, the supervised methods usually cannot remove shadow perfectly (the first and second rows) or keep the non-shadow areas unchanged (the third and fourth rows). Comparatively, TC-GAN effectively removes the shadows and retain the content and illumination in non-shadow areas simultaneously, even if the paired data are not used for training.