Shadow Generation for Composite Image in Real-World Scenes

Image composition (Niu et al. 2021) targets at pasting a foreground object on another background image to produce a composite image (Lin et al. 2018a; Wu et al. 2019; Zhan, Huang, and Lu 2019; Zhan et al. 2020b; Liu et al. 2020). Many issues would significantly degrade the quality of composite images, such as unreasonable location of foreground or inconsistent color/illumination between foreground and background. Previous works attempted to solve one or multiple issues. For example, image blending methods (Pérez, Gangnet, and Blake 2003; Wu et al. 2019; Zhang, Wen, and Shi 2020; Zhang et al. 2021) were developed to blend foreground and background seamlessly. Image harmonization methods (Tsai et al. 2017; Cun and Pun 2020; Cong et al. 2020, 2021) were proposed to address the color/illumination discrepancy between foreground and background. Some other approaches (Chen and Kae 2019; Weng et al. 2020; Zhan, Zhu, and Lu 2019) aimed to cope with the inconsistency of geometry, color, and boundary simultaneously. However, these methods did not consider the shadow effect of inserted foreground on background image, which is the focus of this paper.

Prior works on shadow generation can be divided into two groups: rendering based methods and image-to-image translation methods.

The shadow mask generator $G_{S}$ aims to predict the binary shadow mask $\tilde{\bm{M}}_{fs}$ of the foreground object. We adopt U-Net (Ronneberger, Fischer, and Brox 2015) structure consisting of an encoder $E_{S}$ and a decoder $D_{S}$. To better extract foreground and background information, we split $E_{S}$ into a foreground encoder $E_{FS}$ and a background encoder $E_{BS}$. The foreground encoder $E_{FS}$ takes the concatenation of input composite image $\mathbf{I}_{c}$ and foreground object mask $\bm{M}_{fo}$ as input, producing the foreground feature map $\bm{X}_{f}=E_{FS}(\mathbf{I}_{c},\bm{M}_{fo})$. The background encoder $E_{BS}$ is expected to infer implicit illumination information from background. Considering that the background object-shadow pairs can provide strong illumination cues, we introduce background object-shadow mask $\bm{M}_{bos}$ enclosing all background object-shadow pairs. The background encoder $E_{BS}$ takes the concatenation of $\mathbf{I}_{c}$ and $\bm{M}_{bos}$ as input, producing the background feature map $\bm{X}_{b}=E_{BS}(\mathbf{I}_{c},\bm{M}_{bos})$.

The illumination information in different image regions may vary due to complicated scene geometry and light sources, which greatly increases the difficulty of shadow mask generation (Zhang, Liang, and Wang 2019). Thus, it is crucial to attend relevant illumination information to generate foreground shadow. Inspired by previous attention-based methods (Zhang et al. 2019a; Wang et al. 2018; Vaswani et al. 2017), we use a Cross-Attention Integration (CAI) layer to help foreground feature map $\bm{X}_{f}$ attend relevant illumination information from background feature map $\bm{X}_{b}$.

Firstly, $\bm{X}_{f}\in\mathbb{R}^{H\times W\times C}$ and $\bm{X}_{b}\in\mathbb{R}^{H\times W\times C}$ are projected to a common space by $f(\cdot)$ and $g(\cdot)$ respectively, where $f(\cdot)$ and $g(\cdot)$ are $1\times 1$ convolutional layer with spectral normalization (Miyato et al. 2018). For ease of calculation, we reshape $f(\bm{X}_{b})\in\mathbb{R}^{W\times H\times\frac{C}{8}}$ (resp., $g(\bm{X}_{f})\in\mathbb{R}^{W\times H\times\frac{C}{8}}$) into $\bar{f}(\bm{X}_{b})\in\mathbb{R}^{N\times\frac{C}{8}}$ (resp., $\bar{g}(\bm{X}_{f})\in\mathbb{R}^{N\times\frac{C}{8}}$), in which $N=W\times H$. Then, we can calculate the affinity map between $\bm{X}_{f}$ and $\bm{X}_{b}$:

$\displaystyle\bm{A}=softmax\left(\bar{g}(\bm{X}_{f})\bar{f}(\bm{X}_{b})^{T}\right).$

(1)

With obtained affinity map $\bm{A}$, we attend information from $\bm{X}_{b}$ and arrive at the attended feature map $\bm{X}_{b}^{\prime}$:

$\displaystyle\bm{X}_{b}^{\prime}=v\left(\bm{A}\bar{h}(\bm{X}_{b})\right),$

(2)

where $\bar{h}(\cdot)$ means $1\times 1$ convolutional layer followed by reshaping to $\mathbb{R}^{N\times\frac{C}{8}}$, similar to $\bar{f}(\cdot)$ and $\bar{g}(\cdot)$ in Eqn. 1. $v(\cdot)$ reshapes the feature map back to $\mathbb{R}^{W\times H\times\frac{C}{8}}$ and then performs $1\times 1$ convolution. Because the attended illumination information should be combined with the foreground information to generate foreground shadow mask, we concatenate $\bm{X}_{b}^{\prime}$ and $\bm{X}_{f}$, which is fed into the decoder $D_{S}$ to produce foreground shadow mask $\tilde{\bm{M}}_{fs}$:

$\displaystyle\tilde{\bm{M}}_{fs}=D_{S}([\bm{X}_{b}^{\prime},\bm{X}_{f}]),$

(3)

which is enforced to be close to the ground-truth foreground shadow mask $\bm{M}_{fs}$ by

$\displaystyle\mathcal{L}_{S}=||\bm{M}_{fs}-\tilde{\bm{M}}_{fs}||_{2}^{2}.$

(4)

Although cross attention is not a new idea, this is the first time to achieve foreground-background interaction via cross-attention in shadow generation task.

We design our shadow filling stage based on the illumination model used in  (Shor and Lischinski 2008; Le and Samaras 2019). According to (Shor and Lischinski 2008; Le and Samaras 2019), the value of a shadow-free pixel $I^{lit}(k,i)$ can be linearly transformed from its shadowed value $I^{dark}(k,i)$:

$\displaystyle I^{lit}(k,i)=w^{lit}(k)I^{dark}(k,i)+b^{lit}(k),$

(5)

in which $I(k,i)$ represents the value of the pixel $i$ in color channel $k$ ($k\in R,G,B$). $w^{lit}(k)$ and $b^{lit}(k)$ are constant across all pixels in the umbra area of the shadow. Inversely, the value of a shadowed pixel $I^{dark}(k,i)$ can be linearly transformed from its shadow-free value $I^{lit}(k,i)$:

$\displaystyle I^{dark}(k,i)=w^{dark}(k)I^{lit}(k,i)+b^{dark}(k).$

(6)

To accurately locate the foreground shadow area, we tend to learn a soft shadow matte $\bm{\alpha}$. The value of $\bm{\alpha}$ is 0 in the non-shadow area, 1 in the umbra of shadow area, and varying gradually in the penumbra of shadow area. Then, the target image with foreground shadow can be obtained using the following composition system (see Figure 4):

	$\displaystyle\mathbf{I}_{g}=\mathbf{I}_{c}\circ(\mathbf{1}-{\bm{\alpha}})+\mathbf{I}_{c}^{dark}\circ{\bm{\alpha}},$		(7)
	$\displaystyle\mathbf{I}_{c}^{dark}(k)=w^{dark}(k)\mathbf{I}_{c}(k)+b^{dark}(k),$		(8)

in which $\circ$ means element-wise multiplication, $\mathbf{I}(k)$ represents image $\mathbf{I}$ in color channel $k$, $\mathbf{I}_{c}^{dark}(k)$ is the darkened version of $\mathbf{I}_{c}(k)$ through Eqn. 8. $\mathbf{w}^{dark}=[w^{dark}(R),w^{dark}(G),w^{dark}(B)]$ and similarly defined $\mathbf{b}^{dark}$ are called shadow parameters. Given paired images $\{\mathbf{I}_{c},\mathbf{I}_{g}\}$, the ground-truth shadow parameter $\{\mathbf{w}^{dark},\mathbf{b}^{dark}\}$ for the foreground shadow can be easily calculated by using linear regression (Shor and Lischinski 2008). Specifically, we need to calculate the optimal regression coefficients $\{\mathbf{w}^{dark},\mathbf{b}^{dark}\}$ which regress pixel values $\mathbf{I}_{c}(k,i)$ to $\mathbf{I}_{g}(k,i)$ in the foreground shadow area. The ground-truth shadow parameters of training images can be precomputed before training, but the ground-truth shadow parameters of test images are unavailable in the testing stage. Thus, we learn a shadow parameter predictor $E_{P}$ to estimate $\{\mathbf{w}^{dark},\mathbf{b}^{dark}\}$.

Our $E_{P}$ is implemented as an encoder, which takes the concatenation of composite image ${\mathbf{I}}_{c}$ and predicted shadow mask $\tilde{\bm{M}}_{fs}$ as input to predict the shadow parameters $\{\tilde{\bm{w}}^{dark},\tilde{\bm{b}}^{dark}\}$:

$\displaystyle\{\tilde{\bm{w}}^{dark},\tilde{\bm{b}}^{dark}\}=E_{P}({\mathbf{I}}_{c},\tilde{\bm{M}}_{fs}).$

(9)

$\{\tilde{\bm{w}}^{dark},\tilde{\bm{b}}^{dark}\}$ are supervised with ground-truth shadow parameters $\{\bm{w}^{dark},\bm{b}^{dark}\}$ by regression loss:

$\displaystyle\mathcal{L}_{P}=||\bm{w}^{dark}-\tilde{\bm{w}}^{dark}||_{2}^{2}+||\bm{b}^{dark}-\tilde{\bm{b}}^{dark}||_{2}^{2}.$

(10)

After estimating $\{\tilde{\bm{w}}^{dark},\tilde{\bm{b}}^{dark}\}$, we can get the darkened image $\tilde{\mathbf{I}}_{c}^{dark}(k)=\tilde{w}^{dark}(k)\mathbf{I}_{c}(k)+\tilde{b}^{dark}(k)$ via Eqn. 8. Then, to obtain the final target image, we need to learn a shadow matte $\bm{\alpha}$ for image composition as in Eqn. 7. Our shadow matte generator $G_{M}$ is based on U-Net (Ronneberger, Fischer, and Brox 2015) with encoder $E_{M}$ and decoder $D_{M}$. $G_{M}$ concatenates composite image $\mathbf{I}_{c}$, darkened image $\tilde{\mathbf{I}}_{c}^{dark}$, and predicted shadow mask $\tilde{\bm{M}}_{fs}$ as input, producing the shadow matte $\tilde{\bm{\alpha}}$:

$\displaystyle\tilde{\bm{\alpha}}=G_{M}(\mathbf{I}_{c},\tilde{\mathbf{I}}_{c}^{dark},\tilde{\bm{M}}_{fs}).$

(11)

Finally, based on $\tilde{\mathbf{I}}_{c}^{dark}$, $\mathbf{I}_{c}$, and $\tilde{\bm{\alpha}}$, the target image with foreground shadow can be composed by

$\displaystyle\tilde{\mathbf{I}}_{g}=\mathbf{I}_{c}\circ(\mathbf{1}-\tilde{\bm{\alpha}})+\tilde{\mathbf{I}}_{c}^{dark}\circ\tilde{\bm{\alpha}}.$

(12)

The generated target image is supervised by the ground-truth target image with a reconstruction loss:

$\displaystyle\mathcal{L}_{I}=||\mathbf{I}_{g}-\tilde{\mathbf{I}}_{g}||_{2}^{2}.$

(13)

To the best of our knowledge, we are the first to generate shadow by blending original image and darkened image.

To ensure that the generated shadow mask $\tilde{\bm{M}}_{fs}$ and the generated target image $\tilde{\mathbf{I}}_{g}$ are close to real shadow mask $\bm{M}_{fs}$ and real target image $\mathbf{I}_{g}$ respectively, we design a conditional discriminator $D$ to bridge the gap between the generated triplet $\{\tilde{\bm{M}}_{fs},\tilde{\mathbf{I}}_{g},\bm{M}_{fo}\}$ and the real triplet $\{\bm{{M}}_{fs},\mathbf{I}_{g},\bm{M}_{fo}\}$. The architecture of our conditional discriminator is similar to Patch-GAN (Isola et al. 2017), which takes the concatenation of triplet as input. We adopt the hinge adversarial loss (Miyato and Koyama 2018) as follows,

	$\displaystyle\mathcal{L}_{D}=\mathbb{E}_{\tilde{\bm{M}}_{fs},\tilde{\mathbf{I}}_{g},\bm{M}_{fo}}[\max(0,1+\mathrm{D}(\tilde{\bm{M}}_{fs},\tilde{\mathbf{I}}_{g},\bm{M}_{fo}))]+$
	$\displaystyle\quad\quad\mathbb{E}_{\bm{{M}}_{fs},\mathbf{I}_{g},\bm{M}_{fo}}[\max(0,1-\mathrm{D}(\bm{{M}}_{fs},\mathbf{I}_{g},\bm{M}_{fo})],$
	$\displaystyle\mathcal{L}_{GD}=-\mathbb{E}_{\tilde{\bm{M}}_{fs},\tilde{\mathbf{I}}_{g},\bm{M}_{fo}}[\mathrm{D}(\tilde{\bm{M}}_{fs},\tilde{\mathbf{I}}_{g},\bm{M}_{fo})].$		(14)

The overall optimization function can be written as

$\displaystyle\mathcal{L}=\lambda_{S}\mathcal{L}_{S}+\lambda_{I}\mathcal{L}_{I}+\lambda_{P}\mathcal{L}_{P}+\lambda_{GD}\mathcal{L}_{GD}+\mathcal{L}_{D},$

(15)

where $\lambda_{S}$, $\lambda_{I}$, $\lambda_{P}$, and $\lambda_{GD}$ are trade-off parameters.

The parameters of $\{E_{S},CAI,D_{S},E_{P},E_{M},D_{M}\}$ are denoted as $\theta_{G}$, while the parameters of $D$ are denoted as $\theta_{D}$. Following adversarial learning framework (Gulrajani et al. 2017), we use related loss terms to optimize $\theta_{G}$ and $\theta_{D}$ alternatingly. In detail, $\theta_{D}$ is optimized by minimizing $\mathcal{L}_{D}$. Then, $\theta_{G}$ is optimized by minimizing $\lambda_{S}\mathcal{L}_{S}+\lambda_{I}\mathcal{L}_{I}+\lambda_{P}\mathcal{L}_{P}+\lambda_{GD}\mathcal{L}_{GD}$.

On DESOBA dataset, BOS test set and BOS-free test set are evaluated separately and the comparison results are summarized in Table 1. We can observe that our SGRNet achieves the lowest GRMSE, LRMSE and the highest GSSIM, LSSIM, which demonstrates that our method could generate more realistic and compatible shadows for foreground objects compared with baselines. The difference between the results on BOS test set and BOS-free test set is partially caused by the size of foreground shadow, because BOS-free test images usually have larger foreground shadows than BOS test images as shown in Figure 2. We will provide more in-depth comparison by controlling the foreground shadow size in Supplementary.

For qualitative comparison, we show some example images generated by our SGRNet and other baselines on BOS and BOS-free test images in Figure 5. We can see that our SGRNet can generally generate foreground shadows with reasonable shapes and shadow directions compatible with the object-shadow pairs in background. In contrast, other baselines produce foreground shadows with implausible shapes, or even fail to produce any shadow. Our method can also generate reasonable shadows for BOS-free test images, because the background in BOS-free images could also provide a set of illumination cues (e.g., shading, sky appearance variation) (Lalonde, Efros, and Narasimhan 2012; Zhang et al. 2019b) as discussed in Section 3. More visualization results including the intermediate results (e.g., generated foreground shadow mask, generated darkened image) can be found in Supplementary.

We analyze the impact of loss terms and alternative network designs of our SGRNet on BOS test images from DESOBA dataset. Quantitative results are reported in Table 2.

We visualize some examples produced by different ablated methods and conduct ablation studies on BOS-free test images in Supplementary.

To obtain real composite images, we select test images from DESOBA as background images, and paste foreground objects also from test images at reasonable locations on the background images. In this way, we create $100$ real composite images without foreground shadows for evaluation. Because real composite images do not have ground-truth target images, it is impossible to perform quantitative evaluation. Therefore, we conduct user study on the $100$ composite images for subjective evaluation. The visualization results and user study are left to Supplementary.