Chanel-Orderer: A Channel-Ordering Predictor for Tri-Channel
Natural Images

We first define the following partial order:

$$R\succ G\succ B$$

(1)

which suggests that ideally among the three channels, the red channel $R$ should be placed in the first channel, followed by the green channel $G$ and the blue channel $B$.

Then, given a 3-channel image $\mathcal{I}$ with any of 3-permutations $\pi(\mathcal{S}):=\{I_{1},I_{2},I_{3}\}$, we formulate the model $f$ (parameterized by $\theta$) as a scoring function which outputs the ranking scores for each of the channels independently:

$$s_{1}=f_{\theta}(I_{1}),s_{2}=f_{\theta}(I_{2}),s_{3}=f_{\theta}(I_{3})$$

(2)

These scores are interpreted as the likeness scores that should obey the partial order (1). For example, if the groundtruth suggests $I_{i}\succ I_{j}$ according to the partial order (1), then we should enforce the model to output $s_{i}$ and $s_{j}$ such that $s_{i}>s_{j}$; otherwise, $s_{i}\leq s_{j}$. By modifying the model to predict the probability of $s_{i}>s_{j}$:

$$p_{ij}:=\mathbb{P}(s_{i}>s_{j})=\frac{1}{1+\exp(-g(s_{i}-s_{j})/T)}$$

(3)

we can formulate the ordering prediction problem into three seperate binary classification problems ($s_{1}$ versus $s_{2}$, $s_{1}$ versus $s_{3}$, $s_{2}$ versus $s_{3}$). Ideally, such a predicted probability distribution $p_{ij}$ should get close to the desired probability distribution $y_{ij}$:

$$y_{ij}=\begin{cases}1,&\text{if }I_{i}\succ I_{j}\\ 0,&\text{if }I_{i}\prec I_{j}\\ \frac{1}{2},&\text{otherwise}\\ \end{cases}$$

(4)

In Eq. (3), the scalar $T$ denotes temperature that rescales exponent to $\exp$ and the function $g$ should be an increasing differentiable function with regards to the score difference $\Delta_{ij}:=s_{i}-s_{j}$, e.g. the identity function as the simplest choice. However, we empirically find that the choice of the identity function leads to unstable optimization. In the next section, we show a better choice of $g$ that yields amenable optimization.

Formally, given any $\mathcal{I}$, we minimize the cross entropy loss between the predicted $p_{ij}$ and the groundtruth $y_{ij}$ over all the pairs of comparison (which is inherently a function of $s$ and $y$):

	$\displaystyle\min_{\theta}\mathcal{L}(s,y)$
	$\displaystyle:=\sum_{(i,j)\in\{(1,2),(1,3),(2,3)\}}-y_{ij}\log p_{ij}-(1-y_{ij})\log(1-p_{ij})$		(5)

Plugging $p_{ij}$ and $y_{ij}$ into Eq (2.1) yields

	$\displaystyle\min_{\theta}\mathcal{L}(s,y)$
	$\displaystyle=\sum_{(i,j)\in\{(1,2),(1,3),(2,3)\}}(1-y_{ij})\frac{g(s_{i}-s_{j})}{T}$
	$\displaystyle+\log\left(1+\exp\left(-\frac{g(s_{i}-s_{j})}{T}\right)\right)$		(6)

This section introduces two architectural inductive biases that are incorporated into the implementation of Chanel-Orderer: (1) the choice of $g(\cdot)$ and $T$; (2) the architectural design of the scoring function $f_{\theta}(\cdot)$.

Recall that Theorem 2.1 implies that the larger the value of $s_{i}$ is, the more likely $I_{i}$ should be placed in front among all channels ($i=1,2,3$). By virtue of this implication, we can use $s_{i}$ as the indicator of the channel ordering.

Specifically, given an image $\hat{\mathcal{I}}=[I_{1},I_{2},I_{3}]$ whose channels might be permuted in a wrong order, Chanel-Orderer applies its scoring function $f_{\theta}$ to each of the channels to obtain the scores, respectively: $s_{1}=f_{\theta}(I_{1})$, $s_{2}=f_{\theta}(I_{2})$, $s_{3}=f_{\theta}(I_{3})$. And then label the channel with the largest score among the three as the red channel (Red), label the channel with the smallest score as the blue channel (Blue), and label the third one as the green channel (Green). See Algorithm 2 for the specific Python-like implementation.

In most cases, we rarely encounter a scenerio where a model is expected to tell all $3!=6$ possible permutation orders. Rather, in a typical scenario, an RGB image is often mis-displayed in BGR order. To tackle this particular situation, we slightly modify the proposed Chanel-Orderer for all possible permutations into a model variant that detects RGB against BGR.

We inherit the partial order from (1):

$$R\succ B$$

(12)

which suggests that ideally the red channel $R$ should be ranked ahead of the blue channel $B$ and therefore that RGB is preferable over BGR.

Given a tri-channel image $\mathcal{I}$, similarly as earlier, we first unpacks it into three channels, $I_{1}$, $I_{2}$ and $I_{3}$. Then, we concatenate $I_{1}$ and $I_{2}$ which yields $I_{12}$ and concatenate $I_{1}$ and $I_{3}$ which yields $I_{13}$. After a few operations followed by a global average pooling, the scoring function $f_{\theta}$ is expected to score $I_{12}$ and $I_{13}$ (yielding $s_{12}$ and $s_{13}$, respectively) to determine which ranks ahead of the other. To train the scoring function, a similar ranking loss function as in Eq. (2.1) can be applied. For inference, if $s_{12}>s_{13}$, the given image is predicted as RGB; otherwise, it is predicted as BGR.

In this section, we show our proposed Chanel-Orderer is promising in detecting near-gray images from RGB color images. Near-gray images are images which look monochromatic in general but have a few if not none pixels that are polychromatic (see Figure 3 for some examples). Such images, which often appear in posters or advertisements, are mostly photographed for aesthetic purpose: photographers who make such images use polychromatic imagery to highlight the objects in the images and use monochromatic imagery to render the rest. Prior to Chanel-Orderer, existing methods hinges upon statistic thresholding that are determined in a heuristic manner. Chanel-Orderer, in contrast, is data-driven and learns to predict the ranking scores $s_{1}$, $s_{2}$ and $s_{3}$ whose relative values can inherently be used as indicators to determine whether a given image is polychromatic or monochromatic.

Specifically, given an image $\tilde{I}$, we evaluate the ranking scores $s_{i}=f_{\theta}(\tilde{I}_{i})$, for $i=1,2,3$. And then we evaluate score differences between the three pairs which yields $\Delta_{12}$, $\Delta_{13}$, $\Delta_{23}$. Finally, we determine its monochromatism using the following rule: if $\max_{i,j}{|\Delta_{ij}|}<\tau$ (where $\tau$ is a predefined threshold), we decide it as a near-grayscale image; otherwise, it is decided as a polychromatic image.

We evaluate the proposed Chanel-Orderer on three challenging datasets including SiftFlow [14], PASCAL Context [15] and a customized face dataset referred to as CustoFace thereinafter. The first two benchmarks are used to evaluate the model capability on all-permutation ordering prediction, and the last one is used to evaluate the performance on the detection of RGB against BGR.

SiftFlow [14] includes 2,688 annotated images from a subset of the LabelMe database. The 256 × 256 pixel images are based on 8 different outdoor scenes, among them streets, mountains, fields, beaches, and buildings. All images belong to one of 33 semantic classes. For each test image, we permute its channels to obtain $3!=6$ versions of it.

PASCAL Context [15] is an enhanced version of the PASCAL VOC 2010 object detection challenge, and it provides pixel-level labels for all the training images. The dataset encompasses over 400 classes (which includes the original 20 classes from PASCAL VOC, along with background classes from the segmentation dataset), categorized into three groups: objects, stuff, and hybrid categories. Due to the sparsity of many object categories in the dataset, a subset of 59 frequently occurring classes is commonly chosen for practical use.

CustoFace contains nearly 1,500 face images. All images are $128\times 128$ and contain human aligned faces across various races.

We use total accuracy and accuracies in RGB, RBG, GRB, GBR, BRG and BGR to measure the model performance.

The proposed Chanel-Orderer consists of a U-Net architecture [19] with the four layers of encoders that maps an input into 32-channel, 64-channel, 128-channel and 256-channel sequentially, then with a four layers of decoders that map the encoded feature map back to 128-channel, 64-channel, 32-channel and 1-channel. The intermediate activation functions are ReLUs. The training batch size is set to $48$ and the total training epochs is $100$. The initial learning rate is set to $0.001$ and decays with the factor of $0.98$ Throughout the entire training process, we use the Adam optimizer.

The results from Table 1, Table 2 and Table 3 suggest that Chanel-Orderer consistently outperforms Softmax models in almost all cases. Softmax models cast the channel-ordering prediction into a classification problem whereas Chanel-Orderer tackles this problem in ranking spirit. This further suggests that ranking is more preferable as inductive bias than classification in this particular task. This can also be seen from the training progress: we observe, during training, that Chanel-Orderer converges much faster than Softmax models into smaller loss values, which validates the advantage of inductive biases incorporated into the model.