Name Your Colour For the Task: Artificially Discover Colour Naming via Colour Quantisation Transformer

where $\psi$ and $\theta$ represent the parameters of the CQFormer $g$ and the high-level recognition network $f$, respectively. We jointly optimise $g$ and $f$ to find the optimal parameters $\psi^{\star}$ and $\theta^{\star}$. Additionally, $\mathcal{L}_{P}$ is a perceptual structure loss that is focused on preserving the perceptual structure of the image and will be further explained in detail in Sec. 3.4.

Overall Architecture. An overview of the CQFormer is illustrated in Fig. 2 (a). It consists of two main branches: (1) Annotation Branch, which assigns a quantised colour index to each pixel of the input RGB image, and (2) Palette Branch, which is responsible for generating a suitable colour palette.

Annotation Branch. The first component of Annotation Branch is a UNeXt encoder [44] that generates per-pixel categories. Given the input image $\boldsymbol{x}$, the encoder produces a class activation map $\boldsymbol{x}_{activation}\in\mathbb{R}^{H\times W\times C}$, which contains crucial and semantically rich features. Then, different approaches are used to process the class activation map during testing and training.

(1) Testing stage. As shown by the purple lines in Fig.2 (a), we use the $\boldsymbol{x}_{activation}$ as the input to a Softmax function over $C$ channels, and then apply an $\arg\max$ function to produce a colour index map $M(\boldsymbol{x})\in\mathbb{R}^{H\times W}$:

where $\otimes$ represents matrix multiplication.

(2) Training stage. As depicted by the red lines in Fig.2 (a), since the $\arg\max$ function is not differentiable, we use the Softmax function as a substitute during the training stage. To prevent overfitting, we incorporate a temperature parameter $\tau$ [22] into the Softmax function, pushing the probability map distribution closer to a one-hot vector. This results in the probability map $m_{\tau}(\boldsymbol{x})$ with temperature parameter $\tau$, computed as:

Palette Branch. We locate the representative colours by using an attention-based key-point detection strategy, which is originally designed to use transformer queries to automatically find the key-point location, such as bounding-boxes [5], human poses [31] and colour matrix with gamma [11]. In other words, we reformulate the problem of colour quantisation into a 3D spatial key-point localisation task within the whole RGB colour space.

High-Level Recognition Module. Taking the quantised image ($\bar{\boldsymbol{x}}$ or $\widetilde{\boldsymbol{x}}$) as input, the high-level recognition module predicts the results of object detection or classification.

Intra-cluster Colour Similarity Regularisation. The CQFormer associates each pixel of the input image with a colour index and forms $C$ clusters covering different parts of the WCS colour probability map. To ensure that the pixels within the same cluster are perceptually similar in colour, we propose the intra-cluster colour similarity regularisation $R_{\operatorname{Colour}}$, which is the mean of the colour variance within the $C$ clusters. At first, the centroid colour values of the $C$ clusters $\{\mu_{1},\mu_{2},\dots,\mu_{C}\}$ are calculated. Then, we compute $C$ squared colour distances $\operatorname{d}^{2}_{\operatorname{HSV}}$ between all pixels and their centroid colour value in each cluster. Here, the $\operatorname{d}^{2}_{\operatorname{HSV}}$ is calculated in Munsell HSV colour space as:

where $N_{c}$ is the number of all pixels in $Cluster_{c}$, and $\boldsymbol{x}_{c}[i]$ represents the Munsell HSV value of the $i$-th pixel.

Diversity Regularisation. To encourage the CQFormer to select at least one pixel of all $C$ colours, we adopt the diversity regularisation term $R_{\operatorname{Diversity}}$ proposed by Hou et al. [25]. Diversity is a simple yet efficient metric and serves as an implicit entropy of the assignment distribution. The $R_{\operatorname{Diversity}}$ is calculated as:

Perceptual Similarity Loss. The perceptual similarity loss, denoted by $\mathcal{L}_{\operatorname{Perceptual}}$, is a mean squared error (MSE) loss between the quantised image $\widetilde{\boldsymbol{x}}$ and the input image $\boldsymbol{x}$. It ensures that each item of the colour palette $P(\boldsymbol{x})$ lies in the centre of corresponding colour distribution in the WCS colour probability map (noted by asteroids in Fig. 2 (b)). The $\mathcal{L}_{\operatorname{Perceptual}}$ is calculated as:

Embedding Stage.

(1) Colour Probability Map Embedding. As illustrated in Fig. 3, this embedding forces our CQFormer to match the identical WCS colour probability map of a certain language. At first, we ensure that the CQFormer outputs the same number of quantised colours as the certain human colour system. For each pixel in the input image, we collect its Munsell (hue, value) coordinate and spatial position coordinate $(i,j)$ in the input image. Then we locate this (hue, value) coordinate on the WCS colour probability map of a certain language (Fig. 3 (a)$\rightarrow$(b)$\rightarrow$(c)) to find its corresponding $C$ probability values $p_{\operatorname{Human}}(i,j)\in\mathbb{R}^{C}$ of human colour categories,e.g. 9% for ‘fiNge’, 4% for ‘wOO’, and 87% for ‘nyiE’. After performing the above operations on all pixels of the input image, a set of probability values $\{p_{\operatorname{Human}}(i,j)\}_{(i,j)=(1,1)}^{(H,W)}$ are generated. We arrange each of them according to their spatial position coordinates $(i,j)$ to obtain a new matrix regarded as human language probability map $m_{\operatorname{Human}}(\boldsymbol{x})\in\mathbb{R}^{H\times W\times C}$. Then we use a cross-entropy loss $\mathcal{L}_{CE}$ instead of $\mathcal{L}_{P}$. Finally, for this full embedding solution, we minimise the loss function as:

(2) Central Colour Embedding. Alternatively, we could distil less colour naming information. Here, we only embed representative colours and their $C$ central colour (hue, value) coordinates $\{\mu_{\operatorname{Human},c}\}_{c=1}^{C}$ in the WCS colour probability map of the certain human language (see three asteroids in Fig. 3 (c)). For this embedding solution, we minimise the loss function as:

where we replace the $\mu_{c}$ in $R_{\operatorname{Colour}}$ with aforementioned $\mu_{\operatorname{Human},c}$ and ignore the saturation of Munsell HSV by setting $s_{1}=s_{2}=1$.

Colour Evolution Stage. After the distillation of a certain colour naming system of human language, we evolve the CQFormer to split fine-grained new colour. In this stage, we remove the restriction on the number of quantised colours and minimise different combinations of losses to encourage the CQFormer to evolve more colours. Please refer to Sec. 4.4 for more details.

Datasets. For object detection, we utilise MS COCO [34] dataset, which contains $\sim$118k images with bounding box annotation in 80 categories. Here, we use COCO train2017 set as the train set and use COCO val2017 set as the test set. For classification, we use CIFAR10, CIFAR100 [28], STL10 [10] and tiny-imagenet-200 (Tiny200) [29] dataset. Both CIFAR10 and CIFAR100 contain 60000 images, covering 10 and 100 classes of general objects, respectively. The STL10 consists of 13000 images (5000 for training and 8000 for testing) with 10 classes. The Tiny200 is a subset of ImageNet [13] and contains 200 categories with 110k images. We adopt random crop and random flip for data augmentation.

Evaluation Metrics. For object detection, we report average precision (AP) value. For classification, we report top-1 classification accuracy.

Implement Details.

(1) Upper bound. We utilise the performance of the detector/classifier without additional colour quantisation methods in full-colour space (24-bit) as the upper bound. For object detection, we adopt the Sparse-RCNN [43] detector with ResNet-50 [21] backbone and FPN [33] neck. For classification, we adopt Resnet-18 [21] network.

(2) Training Settings. All colour quantisation experiments are finished at quantisation levels from 1-bit to 6-bit, i.e. $C\in\{2,4,8,16,32,64\}$. We set the temperature parameter $\tau=0.01$ and the $\alpha=1$, $\beta=0.3$ , $\gamma=1$. For detection, only the Resnet-50 backbone of the detector is initialised with Imagenet [13] pre-train weights. We jointly train our CQFormer with Sparse-RCNN for 36 epochs on 4 Tesla V100 GPUs. The batch size is set to 8, and the optimiser is AdamW [36]. The initial learning rate is set to $1.25e^{-6}$, and the learning rate would decay to one-tenth at 24 epochs. For classification, we cascade CQFormer with the Resnet-18 and jointly train them without pre-trained models on a single GeForce RTX A6000 GPU. We employ an SGD optimiser for 60 epochs using the Cosine-Warm-Restart [35] as the learning rate scheduler. A batch size of 128 (STL10 is set to 32), an initial learning rate of 0.05, a momentum of 0.5, and a weight decay of 0.001 are used.

Comparison Methods. We compare with three traditional perception-centred methods: MedianCut [23] , MedianCut+Dither [17] and OCTree [19], and a task-centred CNN-based method ColorCNN [25]. Specifically, for ColorCNN, we adopt the same training strategy as we did for CQFormer. For the traditional methods, we conduct comparative experiments as described in [25].

Results. Fig. 4 presents comparisons to the state-of-the-art methods on the four datasets. Our proposed CQFormer (solid blue line) has a consistent and obvious improvement over all other methods in extremely low-bit colour space (less than 3-bit). Moreover, our CQFormer archives a superior performance than the task-centred method ColorCNN [25] under all colour quantisation levels from 1-bit to 6-bit.

Discussion of classification under a large colour space. Similar to the task-centred ColorCNN, our classification performance is also inferior to the traditional method under a large colour space (greater than 4-bit), despite superior performance on object detection task. As extensively discussed in [25], this is a common characteristic for non-clustering based quantisation.

Settings. This experiment is based on the classification task on CIFAR10 [28] dataset using the Resnet-18 [21] classifier. In the embedding stage, we embed the 1978 Nafaanra three-colour system (Fig. 3 (c)) using the colour probability map embedding and set $C$ as 3. The reason why we choose Nafaanra is analysed in the Supplementary Material. Here we set $\tau=1.0$, and force the fourth colour to be split from light (‘fiNge’), dark (‘wOO’), and warm or red-like (‘nyiE’), respectively. We optimise the loss function $\mathcal{L_{\operatorname{Full-Embedding}}}$ in Eq. 14 for the initial 40 epochs. In the colour evolution stage, we design various combinations of loss function, i.e. only $\mathcal{L}_{M}$, $\mathcal{L}_{M}+R_{\operatorname{Diversity}}$ and $\mathcal{L}_{M}+R_{\operatorname{Colour}}+R_{\operatorname{Diversity}}+\mathcal{L}_{\operatorname{Perceptual}}$, and minimise them separately for the subsequent 20 epochs without the restriction of colour size. Additionally, we also conduct a colour evolution experiment on the task-centred ColorCNN [25] for comparison, which is slightly modified to implement the same embedding and evolution details.

Results and Analysis. The results of colour evolution evaluation is shown in Fig. 5. First, during the embedding stage, the WCS colour probability map generated by the CQFormer (Fig. 5 (a)) is similar to Nafaanra (Fig.3 (c)). Therefore, we successfully embed the colour naming system of human language into the latent space of CQFormer. Second, during the colour evolution stage, as shown in Fig. 5 (b)(c)(d), the CQFormer automatically evolves the fourth colour that is split from dark (‘wOO’) and close to yellow-green under all combinations, matching the basic colour terms theory [27]. Third, although the ColorCNN also evolves the fourth colour close to brown (Fig. 5 (f)), it fails to match the basic colour terms theory [27] since the yellow-green and blue are skipped.