HYouTube: Video Harmonization Dataset

Since constructing video harmonization dataset requires foreground masks and the cost of annotating foreground masks is very high, we build our dataset based on the existing large-scale video object segmentation dataset YouTube-VOS 2018 [17]. YouTube-VOS contains 4453 YouTube video clips and one video clip is annotated with the object masks for one or multiple objects. Each second has 6 frames with mask annotations and we only utilize these annotated frames. Then, for each annotated foreground object in each video clip, if there exist more than 20 consecutive frames containing this foreground object, we save the first 20 consecutive frames with the corresponding 20 foreground masks as one video sample. After that, we remove the video samples with foreground ratio (the area of foreground over the area of the whole frame) smaller than 1% to ensure that the foreground area is in a reasonable range. After the above filtering steps, there are 3194 video samples left.

Based on real video samples, we adjust the appearance of their foregrounds to make them incompatible with backgrounds, producing synthetic composite videos. We have tried different color transfer methods [15, 18, 9, 20] following [4, 16, 7] and 3D color lookup table (LUT) [11, 1] following [8] to adjust the foreground appearance. The color transfer methods [15, 18, 9, 20] need a reference image and adjust the source image appearance based on the reference image appearance, while LUT is [11, 1] a simple array indexing operation to realize color mapping. We observe that applying [15, 18, 9, 20] requires carefully picking reference images, otherwise the transferred foreground may have obvious artifacts or look unrealistic. Thus, we employ LUT to adjust the foreground appearance for convenience. Since one LUT corresponds to one type of color transfer, we can ensure the diversity of the composite videos by applying different LUTs to video samples. Firstly, we collect more than 400 LUTs from the Internet. Secondly, we calculate their pairwise differences. Specifically, we sample 1000 real video frames. For each real frame, we apply all the collected LUTs to transfer its foreground to obtain composite frames and calculate fMSE between every two composite images as the pairwise difference between two LUTs. We average the pairwise differences over all 1000 real video frames as the final pairwise difference between two LUTs. Finally, we select 100 mutually different LUTs in an iterative approach to enlarge the diversity of synthesized composite videos. In particular, in each iteration, we find two closest LUTs from the remaining LUTs and remove one of them. This step is repeated until there are 100 LUTs left. Thus, we get 100 candidate LUTs with the largest mutual difference, which are used to transfer the foregrounds of real video samples.

The process of generating composite video samples is illustrated in Figure 1. Given a video sample, we first select a LUT from 100 candidate LUTs randomly to transfer the foreground of each frame. Lookup table (LUT) records the input color and the corresponding output color, so one LUT corresponds to one color mapping function $f$. LUT has been applied in a variety of computer vision tasks. An LUT is a 3D lattice in the RGB space and each dimension corresponds to one color channel (e.g., red). LUT consists of $V=(B+1)^{3}$ entries by uniformly discretizing the RGB color space, where $B$ is the number of bins in each dimension (we set $B=32$ following the convention in image processing field). Each entry $v$ in the LUT has an indexing color $\mathbf{c}^{\prime}_{v}=(r^{\prime}_{v},g^{\prime}_{v},b^{\prime}_{v})$ and its corresponding output color $\tilde{\mathbf{c}}^{\prime}_{v}=(\tilde{r}^{\prime}_{v},\tilde{g}^{\prime}_{v},\tilde{b}^{\prime}_{v})$. The color transformation process based on LUT has two steps: look up and trilinear interpolation. Specifically, given a color value, we first look up its eight nearest entries in the LUT, and then interpolate its transformed value based on eight nearest entries via trilinear interpolation.

The transferred foregrounds and the original backgrounds form the composite frames, and the composite frames form composite video samples. Following [4], we set some rules to filter out unqualified composite video samples: 1) The transferred foreground should be obviously incompatible with the background; 2) Although the transferred foreground looks incompatible with the background, the transferred foreground itself should look realistic; 3) The albedo of the foreground should remain the same after color transfer. For example, transferring a red car to a blue car is not meaningful for image harmonization [4]. Given a real video sample, if the obtained composite video sample after applying one LUT does not satisfy the above criteria, we will randomly choose another LUT again and repeat the transfer process until the obtained composite video sample satisfies the above criteria.

We name our constructed video harmonization dataset as HYouTube. HYouTube dataset includes 3194 pairs of synthetic composite video samples and real video samples. Each video sample contains 20 consecutive frames with the foreground mask for each frame. The numbers of composite video samples created using different LUTs are shown in the left subfigure in Figure 2. We can see that all 100 LUTs have been used, but some LUTs are more frequently used because they are suitable for more real video samples. The average fMSE between composite video samples and ground-truth video samples for different LUTs is shown in the right subfigure in Figure 2. It can be seen that the average fMSE using different LUTs is quite different, which proves the diversity of different LUTs to some extent. Finally, we show some example pairs of composite video samples and real video samples in our HYoutube dataset in Figure 3.