A Benchmark for Temporal Color Constancy

The rooted phone saves the frames as unprocessed 16-bit 3648$\times$2736 Bayer pattern images. High-resolution frame transfer from the ISP memory to the mobile phone storage memory requires special functionality that limits the practical capture frame rate to 1-3 frames per second (FPS).

To resolve the illuminant color ground truth the shot frame scenes need to be captured with a color calibration target installed into the scene. For example, in the Gehler-Shi dataset there is a Macbeth color checker calibration target visible in the images. In the SFU Gray Ball dataset a gray ball calibration target is attached to the video camera and is therefore visible in all captured frames. In our dataset we wanted to avoid using visible targets since they may unintentionally convey information to the learning-based methods even if they are masked in the training and test sets.

Similar to SFU Gray Ball we used a gray surface calibration target, SpyderCube (Figure 1), which is put into the shot scene instantly after the shot.²²2The target is always at the image center so that color shading has minimal effect on the ground truth. The students were instructed to take one shot of the calibration target in the location which was the main target or location in their photograph. The captured sequences contain 3-17 frames depending on the viewfinder duration (Figure 1). The SpyderCube object contains two neutral 18% gray surfaces, from which the one that better reflects the casting illumination was annotated and used to compute the ground truth illumination color. The ground truth was verified by manually checking all sequences using the ground truth correction. In total, 600 sequences were recorded and verified during different times of day, in various indoor and outdoor locations and in various weathers during the time period of October 2019 to January 2020.

In the dataset project page we provide linear demosaiced images in the PNG format with the pixel values normalized to $[0,1]$, with a black level of zero and with no saturated pixels. The format correspond to that of Gehler-Shi dataset which is a popular evaluation set in color constancy literature. The black level of the specific camera sensor and device is $256$ and the saturation level is at $4095$. The final RGB images are of the resolution 1824$\times$1368.

We profile the distributions of ground truth chromaticity values of several mainstream color constancy benchmarks (Gehler-Shi, NUS 8-camera, Cube+ and SFU Gray Ball datasets) in the top-left inset of Figure 2, while we show that of the new Temporal Color Constancy dataset (TCC benchmark) in the top-right position. Our chromacity distributions are similar to the popular Gehler-Shi, NUS and Cube+ datasets.³³3Note that SFU Gray Ball distribution is larger than others since the data was captured with a high-end Sony VX-2000 video camera that has separate sensors for each color channel and therefore less spectral cross-talk and better channel separation. The small spatial shifts between the datasets are mainly due to different cameras used in the datasets.

In the bottom left of Figure 2 we draw the histogram of sequence lengths in the TCC benchmark. The mean length is $7.3$, median $7.0$ and mode is $8.5$. The bottom-right inset of Figure 2 shows the correlation coefficient between the sequence lengths and the ground truth vectors, which indicates that there is no clear correlation between the sequence length and the global illumination.

The conventional single-frame methods are designed to estimate the illuminant color from a single image - the shot frame. However, it is straightforward to extend the single-frame methods to the multi-frame setting. A single-frame method is executed on every frame and the per frame estimates are combined using a suitable statistical tool such as the moving average. In the following we introduce temporal extensions of the SoTA statistical and learning-based methods.

In the following, we propose a strong baseline for temporal color constancy. The baseline is a deep network architecture (TCC-Net) inspired by the RCC-Net in [9], but with the following significant improvements: 1) a more powerful backbone network for the semantic feature extraction, 2) 2D LSTM that provides more effective spatial recurrent information and 3) support for variable length sequences. The overall architecture is described in Fig. 3. TCC-Net adopts the two CNN+LSTM branch structure from RCC-Net. The first branch, the temporal branch, processes the image sequence, and the second branch, the shot frame branch, processes a pseudo zoom-out sequence in the shot frame. In TCC-Net the both branches are based on a novel 2D LSTM that produces spatio-temporal information which are merged into a single RGB vector at the end of the processing pipeline.

The backbone feature extraction network of RCC-Net (VGG-Net or AlexNet) is replaced with SqueezeNet [22] in TCC-Net. In a recent architecture for computational color constancy, FC4 [8], the SqueezeNet [22] was found superior and this was verified by our experiments (see Section 5.3). Following [8], we keep all layers up to the last convolution layer of SqueezeNet which outputs a 512-channel 2D feature map.

The second improvement is to adopt a 2D LSTM to temporally process sequences and learn a 2D spatial-temporal illumination feature map. We refer to the ordinary LSTM used by RCC-Net as “1DLSTM” due to the fact that its memory cells and the hidden states are encoded as 1D vectors. Although several 1DLSTMs can be stacked to learn more complex sequence-to-sequence mapping, the nature of 1DLSTM hinders its representative power for spatial information. 2DLSTM, introduced in [23], extends 1DLSTM to 2D space by using convolutional structures in both input-to-state and state-to-state transitions. Combining these changes, we have an end-to-end deep network which predict spatial illumination. To get the global estimate vector, averaging (or more advanced manipulation, e.g. confidence weighted averaging in [8]) is applied.

TCC-Net provides native support to varying-length input. This is implemented by the dynamic computational graph feature supported in PyTorch. In contrast, RCC-Net supports only a pre-defined and fixed length sequences (3 or 5 frames in the original paper). With Nvidia GTX 1080ti the processing speed of TCC-Net is 6 ms per frame (only the network operations).

For better understanding of the network parameters we present the key equations implemented in the TCC-Net architecture. For simplicity, the equations are given only for one branch, but the both branches share the similar stages. Given an input sequence $\{I_{1},\ldots,I_{\textit{len}}\}$ and the SqueezeNet backbone $F_{s}$ TCC-Net proceeds as

where $i_{t},f_{t},o_{t}$ are 3D tensors and refer to the input, forget, and output gates of 2D-LSTM. “$\ast$” denotes convolution and “$\circ$” Hadamard product. 2D-LSTM has two parameters: the convolution kernel size $K$ (a larger value corresponds to faster illumination variations) and the output channel size $H$ of the convolution filter (corresponds to hidden channels of 1D-LSTM). Ablation study of the both parameters is provided in Section 5. Figure 4 visualizes the workings of the TCC-Net pipeline.

The results for various single-frame static and learning-based methods (see the related work section), their temporal extensions (Section 4.1), the current temporal state-of-the-art (RCC-Net) [9] and our temporal baseline (Section 4.2) are shown in Table 1. The results demonstrate that the recent deep learning based methods (FC⁴ and C4) and the convolutional CC (FFCC) are clearly superior to the conventional static and learning-based methods. These methods improve the performance over the whole error distribution, i.e. both the easy and difficult test samples. On our dataset the previous temporal state-of-the-art, RCC-Net [9], is slightly inferior to the best single-frame methods C4 and FFCC.

The temporal extension of GI [6], T.GI, improves its results. On the contrary, T.FFCC, referred to as “temporal smoothing” in [13], is inferior to its single-frame version. The Kalman filter extension of FFCC provides smoother change of the illuminant estimates over the frames, but the accuracy is worse than the non-smoothed estimates. We also test Prinet et al. [16] and it achieves 7.51 mean error due to its assumption that the illumination remains constant over time.

The proposed TCC-Net (Model G in Table 3) obtains the best performance on all error measures and improves performance on the both easy and difficult cases. As compared to the previous state-of-the-art, RCC-Net, the performance improvement is over 35% in the mean error and and 43% in the median error. Considering the fact that end-users are more sensitive to large estimation errors [11] and $\leq 3.0^{\circ}$ is generally considered as the sufficient accuracy, then W25% error of the TCC-Net (4.84) is closest to the practical use among all tested methods.

In Figure 5 are examples of color-corrected images with various methods. The first two examples demonstrate easy cases from outdoors where all methods perform comparably well. The third and fourth examples represent typical view finder sequences toward a target which itself does not provide visually-rich clue for inferring the illumination color. In these sequences the two temporal methods, RCC-Net and TCC-Net, provide the best results since they effectively exploit cues from the view finder frames. The last example is a difficult case where the shot frame is a closeup of a tinted fabric material which can be of any plausible color. For the fifth sequence only the proposed TCC-Net provides an accurate estimate.

To validate the findings in the previous experiment with the new TCC-benchmark, we replicated the experiments in Qian et al.  [9], using their metrics (the mean, median, worst 5% and maximum errors) and the SFU Gray Ball dataset. The results are collected to Table 2 (cf. Table 1 in [9]).

On the temporal version of the SFU Gray Ball dataset, the proposed TCC-Net again outperforms the RCC-Net [9], with a clear margin. The difference of these two methods is particularly evident on the hardest cases as TCC-Net obtains more than 40% lower error on the both worst 5% and the maximum error metrics.

Results with different components and parameter settings of TCC-Net are given in Table 3 and briefly discussed below.