Cross-Camera Convolutional Color Constancy

The goal of computational color constancy is to emulate the human visual system’s ability to constantly perceive object colors even when they are observed under different illumination conditions. In many contexts, this problem is equivalent to the practical problem of automatic white balance—removing an undesirable global color cast caused by the illumination in the scene, thereby, making it appear to have been imaged under a white light (see Figure 1). White balance does not only affect the quality of photographs but also has an impact on the accuracy of different computer vision tasks [4]. On modern digital cameras, automatic white balance is performed for all captured images as an essential part of the camera’s imaging pipeline.

Color constancy is a challenging problem, because it is fundamentally under-constrained: an infinite family of white-balanced images and global color casts can explain the same observed image. Color constancy is, therefore, often framed in terms of inferring the most likely illuminant color given some observed image and some prior knowledge of the spectral properties of the camera’s sensor.

One simple heuristic applied to the color constancy problem is the “gray-world” assumption: that colors in the world tend to be neutral gray and that the color of the illuminant can, therefore, be estimated as the average color of the input image [18]. This gray-world method and its related techniques have the convenient property that they are invariant to much of the spectral sensitivity differences among camera sensors and, therefore, very well-suited to the cross-camera task. If camera A’s red channel is twice as sensitive as camera B’s red channel, then a scene captured by camera A will have an average red intensity that is twice that of the scene captured by camera B, and so gray-world will produce identical output images (though this assumes that the spectral response of A and B are identical up to a scale factor, which is rarely the case in practice). However, current state-of-the-art learning-based methods for color constancy rarely exhibit this property, because they often learn things like the precise distribution of likely illuminant colors (a consequence of black-body illumination and other scene lighting regularities) and are, therefore, sensitive to any mismatch between the spectral sensitivity of the camera used during training and that of the camera used at test time [3].

Because there is often significant spectral variation across camera models (as shown in Figure 2), this sensitivity of existing methods is problematic when designing practical white-balance solutions. Training a learning-based algorithm for a new camera requires collecting hundreds, or thousands, of images with ground-truth illuminant color labels (in practice: images containing a color chart), a burdensome task for a camera manufacturer or platform that may need to support hundreds of different camera models. However, the gray-world assumption still holds surprisingly well across sensors—if given several images from a particular camera, one can do a reasonable job of estimating the range of likely illuminant colors (as can also be seen in Figure 2).

In this paper, we propose a camera-independent color constancy method. Our method achieves high-accuracy cross-camera color constancy through the use of two concepts: First, our system is constructed to take as input not just a single test-set image, but also a small set of additional images from the test set, which are: (i) arbitrarily-selected, (ii) unlabeled, (iii) and not white balanced. This allows the model to calibrate itself to the spectral properties of the test-time camera during inference. We make no assumptions about these additional images except that they come from the same camera as the “target” test set image and they contain some content (not all black or white images). In practice, these images could simply be randomly chosen images from the photographer’s “camera roll”, or they could be a fixed set of ad hoc images of natural scenes taken once by the camera manufacturer—because these images do not need to be annotated, they are abundantly available. Second, our system is constructed as a hypernetwork [34] around an existing color constancy model. The target image and the additional images are used as input to a deep neural network whose output is the weights of a smaller color constancy model, and those generated weights are then used to estimate the illuminant color of the target image.

Our system is trained using labeled (and unlabeled) images from multiple cameras, but at test time our model is able to look at a set of (unlabeled) test set images from a new camera. Our hypernetwork is able to infer the likely spectral properties of the new camera that produced the test set images (much as the reader can infer the likely illuminant colors of a camera from only looking at aggregate statistics, as in Figure 2) and produce a small model that has been dynamically adapted to produce accurate illuminant estimates when applied to the target image. Our method is computationally fast and requires a low memory footprint while achieving state-of-the-art results compared to other camera-independent color constancy methods.

There is a large body of literature proposed for illuminant color estimation, which can be categorized into statistical-based methods (e.g., [18, 17, 65, 25, 40, 68, 32, 20, 60]) and learning-based methods (e.g., [26, 31, 24, 16, 62, 30, 53, 15, 12, 57, 66, 37, 13, 55, 76]). The former rely on statistical-based hypotheses to estimate scene illuminant colors based on the color distribution and/or spatial layout of the input raw image. Such methods are usually simple and efficient, but they are less accurate than the learning-based alternatives.

Learning-based methods, on the other hand, are typically trained for a single target camera model in order to learn the distribution of illuminant colors produced by the target camera’s particular sensor [3, 45, 29]. The learning-based methods are typically constrained to the specific, single camera use-case, as the spectral sensitivity of each camera sensor significantly alters the recorded illuminant and scene colors, and different sensor spectral sensitivities change the illuminant color distribution for the same set of scenes [38, 73]. Such camera-specific methods cannot accurately extrapolate beyond the learned distribution of the training camera model’s illuminant colors [3, 60] without tuning/re-training or pre-calibration [50].

Recently, few-shot and multi-domain learning techniques [75, 55] have been proposed to reduce the effort of re-training camera-specific learned color constancy models. These methods require only a small set of labeled images for a new camera unseen during training. In contrast, our technique requires no ground-truth labels for the unseen camera, and is essentially calibration-free for this new sensor.

Another strategy has been proposed to white balance the input image with several illuminant color candidates and learn the likelihood of properly white-balanced images [35]. Such a Bayesian framework requires prior knowledge of the target camera model’s illuminant colors to build the illuminant candidate set. Despite promising results, these methods, however, all require labeled training examples from the target camera model: raw images paired with ground-truth illuminant colors. Collecting such training examples is a tedious process, as certain conditions must be satisfied—i.e., for each image to have a single uniform lighting and a calibration object to be present in the scene [20].

An additional class of work has sought to learn sensor-independent color constancy models, circumventing the need to re-train or calibrate to a specific camera model. A recent quasi-unsupervised approach to color constancy has been proposed, which learns the semantic features of achromatic objects to help build a model robust to differing camera sensor spectral sensitivities [14]. Another technique proposes to learn an intermediate “device independent” space before the illuminant estimation process [3]. The goal of our method is similar, in that we also propose to learn a color constancy model that works for all cameras, but neither of these previous sensor-independent approaches leverages multiple test images to reason about the spectral properties of the unseen camera model. This enables our method to outperform these state-of-the-art sensor-independent methods across diverse test sets.

Though not commonly applied in color constancy techniques, our proposal to use multiple test-set images at inference-time to improve performance is a well-explored approach across machine learning. The task of classifying an entire test set as accurately as possible was first described by Vapnik as “transductive inference” [69, 39]. Our approach is also closely related to the work on domain adaptation [22, 63] and transfer learning [58], both of which attempt to enable learning-based models to cope with differences between training and test data. Multiple sRGB camera-rendered images of the same scene have been used to estimate the response function of a given camera in the radiometric calibration literature [33, 42]. In our method, however, we employ additional images to learn to extract informative cues about the spectral sensitivity of the camera capturing the input test image, without needing to capture the same scene multiple times.

We call our system “cross-camera convolutional color constancy” (C5), because it builds upon the existing “convolutional color constancy” (CCC) model [12] and its successor “fast Fourier color constancy” (FFCC) [13], but embeds them in a multi-input hypernetwork to enable accurate cross-camera performance. These CCC/FFCC models work by learning to perform localization within a log-chroma histogram space, such as those shown in Figure 2.

Here, we present a convolutional color constancy model that is a simplification of those presented in the original work [12] and its FFCC follow-up [13]. This simple convolutional model will be a fundamental building block that we will use in our larger neural network. The image formation model behind CCC/FFCC (and most color constancy models) is that each pixel of the observed image is assumed to be the element-wise product of some “true” white-balanced image (or equivalently, the observed image if it were imaged under a white illuminant) and some illuminant color:

where $\bm{\mathrm{c}}^{(k)}$ is the observed color of pixel $k$, $\bm{\mathrm{w}}^{(k)}$ is the true color of the pixel, and $\bm{\mathrm{\ell}}$ is the color of the illuminant, all of which are 3-vectors of RGB values. Color constancy algorithms traditionally use the input image $\{\bm{\mathrm{c}}^{(k)}\}$ to produce an estimate of the illuminant $\hat{\bm{\mathrm{\ell}}}$ that is then divided (element-wise) into each observed color to produce an estimate of the true color of each pixel, $\{\hat{\bm{\mathrm{w}}}^{(k)}\}$.

CCC defines two log-chroma measures for each pixel, which are simply the log of the ratio of two color channels:

As noted by Finlayson, this log-chrominance representation of color means that illuminant changes (i.e.  element-wise scaling by $\bm{\mathrm{\ell}}$) can be modeled simply as additive offsets to this $uv$ representation [23]. We then construct a 2D histogram of the log-chroma values of all pixels:

This is simply a histogram over all $uv$ coordinates of size ($64\times 64)$ written out using Iverson brackets, where $\epsilon$ is the width of a histogram bin, and where each pixel is weighted by its overall brightness under the assumption that bright pixels provide more actionable signal than dark pixels. As was done in FFCC, we construct two histograms: one of pixel intensities, $N_{0}$, and one of gradient intensities, $N_{1}$. The latter is constructed analogously to Equation 3.

These histograms of log-chroma values exhibit a useful property: element-wise multiplication of the RGB values of an image by a constant results in a translation of the resulting log-chrominance histograms. The core insight of CCC is that this property allows color constancy to be framed as the problem of “localizing” a log-chroma histogram in this $uv$ histogram-space [12]—because every $uv$ location in $N$ corresponds to a (normalized) illuminant color, $\bm{\mathrm{\ell}}$, the problem of estimating $\bm{\mathrm{\ell}}$ is reducible (in a computability sense) to the problem of estimating a $uv$ coordinate. This can be done by discriminatively training a “sliding window” classifier much as one might train, say, a face-detection system: the histogram is convolved with a (learned) filter and the location of the argmax is extracted from the filter response, and that argmax corresponds to $uv$ value that is (the inverse of) an estimated illumination location.

We adopt a simplification of the convolutional structure used by FFCC [13]:

where $\{F_{i}\}$ and $B$ are filters and a bias, respectively, which have the same shape as $N_{i}$. Each histogram, $N_{i}$, is convolved with each filter, $F_{i}$, and summed across channels (a “conv” layer). Then, the bias, $B$, is added to that summation, which collectively biases inference towards $uv$ coordinates that correspond to common illuminants, such as black body radiation.

As was done in FFCC, this convolution is accelerated through the use of FFTs, though, unlike FFCC, we use a non-wrapped histogram and thus non-wrapped filters and bias. This avoids the need for the complicated “de-aliasing” scheme used by FFCC which is not compatible with the convolutional neural network structure that we will later introduce.

The output of the softmax, $P$, is effectively a “heat map” of what illuminants are likely, given the distribution of pixel and gradient intensities reflected in $N$ and in the prior $B$, from which, we extract a “soft argmax” by taking the expectation of $u$ and $v$ with respect to $P$:

Equation 5 is equivalent to estimating the mean of a fitted Gaussian, in the $uv$ space, weighted by $P$. Because the absolute scale of $\bm{\mathrm{\ell}}$ is assumed to be irrelevant or unrecoverable in the context of color constancy, after estimating $(\hat{\ell}_{u},\hat{\ell}_{v})$, we produce an RGB illuminant estimate, $\hat{\bm{\mathrm{\ell}}}$, that is simply the unit vector whose log-chroma values match our estimate:

A convolutional color constancy model is then trained by setting $\{F_{i}\}$ and $B$ to be free parameters which are then optimized to minimize the difference between the predicted illuminant, $\hat{\bm{\mathrm{\ell}}}$, and the ground-truth illuminant, $\bm{\mathrm{\ell}}^{*}$.

In all experiments we used $384\!\times\!256$ raw images after applying the black-level normalization and masking out the calibration object to avoid any “leakage” during the evaluation. Excluding histogram computation time (which is difficult to profile accurately due to the expensive nature of scatter-type operations in deep learning frameworks), our method runs in $\sim$7 milliseconds per image on a NVIDIA GeForce GTX 1080, and $\sim$90 milliseconds on an Intel Xeon CPU Processor E5-1607 v4 (10M Cache, 3.10 GHz). Because our model exists in log-chroma histogram space, the uncompressed size of our entire model is $\sim$2 MB, small enough to easily fit within the narrow constraints of limited compute environments such as mobile phones.

We have presented C5, a cross-camera convolutional color constancy method. By embedding the existing state-of-the-art convolutional color constancy model (CCC) [12, 13] into a multi-input hypernetwork approach, C5 can be trained on images from multiple cameras, but at test time synthesize weights for a CCC-like model that is dynamically calibrated to the spectral properties of the previously-unseen camera of the test-set image. Extensive experimentation demonstrates that C5 achieves state-of-the-art performance on cross-camera color constancy for several datasets. By enabling accurate illuminant estimation without requiring the tedious collection of labeled training data for every particular camera, we hope that C5 will accelerate the widespread adoption of learning-based white balance by the camera industry.