Exploring Color Invariance through Image-Level Ensemble Learning

In solving the problem of color deviation, the early work Li et al. (2014) used the filter and the maximum grouping layer to learn the illumination transformation, divided the pedestrian image into more small pieces to calculate the similarity, and uniformly handled the problems of misalignment, occlusion and illumination variation under the deep neural network. Liao et al. Liao et al. (2015) performed pre-processing before feature extraction and used multiscale Retinex algorithm to enhance the color information of light shaded regions to improve the color changes caused by lighting condition changes.

Many data augmentation methods have been proposed, such as random cropping, flipping, which are widely used in computer vision. Different methods address distinct issues. CutMix Yun et al. (2019) replaces one patch of an image with a patch from another image, it helps improve the robustness and generalization of deep learning models by encouraging them to learn from mixed and diverse training samples. Autoaugment Cubuk et al. (2019) is a automated data augmentation strategy, which incorporates a series of data augmentation such as image shear transformations, color adjustments, brightness/contrast adjustments, etc.. Co-Mixup Kim et al. (2021) combines CutMix and Mixup Guo et al. (2019) methods to enhance model performance and generalization in deep learning tasks. Random erasing or cutout Zhong et al. (2020); DeVries and Taylor (2017) adds noise block to the image to regularize the network, while it helps to solve the occlusion problem.

Please note that our approach differs fundamentally in motivation and effect from the methods mentioned above. For instance, Random erasing primarily aims to address the issue of decreased generalization performance when the target is occluded. Although this method appears to eliminate color, it actually directly disrupts the structural information of the image. Therefore, the model cannot learn color invariance from relevant parts with or without color. In fact, when the camera style changes, it may even degrade the model’s original performance. This observation is further validated in our cross-domain experiments.

With the increasing maturity of GANs, GANs-based approaches for data augmentation have become an active research field.

In ReID, the goal of GANs methods Gu et al. (2022); Zheng et al. (2019); Zhong et al. (2018b); Liu et al. (2018) is to mitigate the effect of clothing change of pedestrians, color deviation or human-pose variation, and to improve the robustness of the model by learning the invariant features from the variation of the input. The appearance details and the emphases generated by different GANs-based methods are also different, but their goal is all to compensate for the difference between the source and target domains. For example, CamStyle Zhong et al. (2018b) generates new data for transferring different camera styles to learn invariant features between different cameras to increase the robustness of the model to camera style changes. CycleGAN Zhu et al. (2017) was applied in Deng et al. (2018); Zhong et al. (2019) to transfer pedestrian image styles from one dataset to another. StarGAN Choi et al. (2018) was used by Zhong et al. (2018a) to generate pedestrian images with different camera styles. Wei et al. Wei et al. (2018) proposed PTGAN to achieve pedestrian image transfer across different ReID datasets. It uses semantic segmentation to extract foreground masks to assist style transfer, and converts the background into the desired style of the dataset while keeping the foreground unchanged. Different from global style transfer, DGNet Zheng et al. (2019) utilized GANs to transfer clothing color among different pedestrians to generate more diverse data to reduce the impact of color changes on the model, which effectively improves the generalization ability of the model. CCFA Han et al. (2023) is a variant of DGNet. In contrast to DGNet, which primarily emphasizes learning color invariance, CCFA places greater emphasis on addressing the challenge of insufficient diversity in clothing styles within the training data. Therefore, in the experiments conducted in this paper, our primary focus is to compare with DGNet.

Up to now, GAN-based methods in ReID are considered the optimal choice for enhancing the model’s robustness to color variations. By exploring color invariance in ensemble learning, we now present a novel solution in our research.

In the data loading, it randomly samples M images of per person and K identities to constitute a training batch, which size is $B=M\times K$. The set is denoted as $x^{v}=\{x_{i}^{v}|i=1,2,...,M\times K\}$.

The proposed method randomly performs global grayscale transformation on the training batch with a probability, and then inputs the processed images into the model for training. This grayscale transformation process can be defined as:

where $t(\bullet)$ is the grayscale image conversion function, which is implemented pixel-by-pixel accumulation calculations on the R, G, and B channels of the original visible RGB image.

The local color erasing for each visible image $x_{i}^{v}$ can be achieved by the following equations:

the function $LT(\bullet)$ can be represented as:

$RandPosition(\bullet)$ is used to generate a random rectangle in the image, the function of $LT(\bullet)$ is to replace the pixels at the rectangular position in the image $x_{i}^{g}$ with the pixels at the corresponding position in the image $x_{i}^{v}$. And $x_{i}^{lg}$ is the sample after local grayscale transformation.

In the process of model training, we apply a local grayscale transformation function $LT(\bullet)$, randomly to the training batch with a certain probability. For an image $x_{i}^{v}$ within a batch, $p_{r}$ represents the probability of performing $LT(\bullet)$ on the image $x_{i}^{v}$. During this process, a rectangular region in the image is randomly selected and replaced with pixels from the corresponding rectangular region in the grayscale image. This introduces various levels of grayscale information into the training images. This process generates training images with diverse levels of grayscale while preserving the structural integrity of the objects. For specific details on the random selection of rectangular regions, please refer to random erasing Zhong et al. (2020). We adopt a similar process and parameter settings.

Assuming this task is to approximate a function $f:Q^{m}\rightarrow Y$ using an ensemble composed of $N$ component neural networks, where $Y$ is the set of class labels, and the prediction of component networks are combined by majority voting, in which each component network votes for a class, and the class label with the largest number of votes is obtained as the output of the ensemble. For the convenience of discussion, assuming that $Y$ contains only two class labels. The set of two class labels is usually denoted as {0,1}, but for the convenience of derivation, here use {-1, +1} instead of {0,1} to represent the class label. So the function to be approximated is $f:Q^{m}\rightarrow\{-1,+1\}$. Nevertheless, please note that the following derivation can also be extended to the case where $Y$ contains more than two class labels.

Assuming there are $k$ samples, the expected output $O=[o_{1},o_{2},…,o_{k}]^{T}$, where $d_{j}$ denotes the expected output on the $j$-th instance, and the actual output of the $i$-th component neural network is $F_{i}=[f_{i1},f_{i2},…,f_{ik}]^{T}$ where $f_{ij}$ denotes the actual output of the $i$-th component network on the $j$-th sample. $O$ and $F_{i}$ satisfy that $o_{j}$ $\in\{-1,+1\}(j=1,2,…,m)$ and $f_{ij}$$\in\{-1,+1\}(i=1,2,…,N;j=1,2,…,k)$, respectively. If the actual output of the $i$-th component network on the $j$-th sample is correct according to the expected output then $f_{ij}o_{j}=+1$, otherwise $f_{ij}o_{j}=-1$. Thus the generalization error of the $i$-th component neural network on those $k$ sample is:

where $Error(\bullet)$ is a function defined as:

Here we introduce a vector $S=[S_{1},S_{2},…,S_{k}]^{T}$ where $S_{j}$ denotes the sum of the actual output of all the component neural networks on the $j$-th sample:

Then the output of the neural network ensemble on the j-th sample is:

where $\hat{f}_{j}\in\{-1,0,+1\}(j=1,2,…,k)$, and $Sgn(\bullet)$ is a function defined as:

If the output of the ensemble on the $j$-th sample is correct according to the expected output then $\hat{f_{j}}o_{j}=+1$. If it is wrong then $\hat{f}_{j}o_{j}=-1$. Otherwise, $\hat{f}_{j}o_{j}=0$, which means that there is a tie on the j-th sample. It means that half component networks vote for +1 while the other half networks vote for -1. Thus the generalization error of the ensemble is:

Here suppose that the $e$-th component neural network is trained using grayscale images. Then the output of the new ensemble on the $j$-th instance is:

and the generalization error of the new ensemble is:

Assuming a certain quantity of networks with color information deviation will not impact the overall performance of the neural network, and neglecting color information may lead to improved retrieval results for certain examples.

From Eq. (10) and Eqn. (12) we can derive that if Eqn. (13) is satisfied then $\hat{E}$ is not smaller than $\hat{E^{{}^{\prime}}}$, and then:

Eqn. (14) indicates that the ensemble including $e$-th component neural network which is trained using grayscale images is better than the one no including, as shown in Fig. 3.

Now, it can be reach the conclusion that in the context of classification, when a number of neural networks are available, ensembling part of them to fit the color features may be better than ensembling all of them.

We evaluate our method on three ReID datasets, including Market-1501 Zheng et al. (2015), DukeMTMC Zheng et al. (2017), MSMT17 Wei et al. (2018). These three datasets are among the most representative and extensively utilized in ReID research. Together, they encompass multiple seasons, time periods, high-definition, and low-definition cameras, providing rich scenes, backgrounds, and intricate lighting variations.

Following existing works Zheng et al. (2015), we employ Rank-k precision and Cumulative Matching Characteristics (CMC) and mean Average Precision (mAP) as evaluation metrics. Rank-1 represents the average accuracy of the top-ranked result corresponding to each cross-modality query image. mAP represents the mean average accuracy, where the query results are sorted based on similarity, and the closer the correct result is to the top of the list, the higher the precision. As part of a ReID system, re-ranking Zhong et al. (2017) technology is typically employed to reorganize the initial retrieval results, aiming to more accurately reflect the similarity between images.

DustProj consists of 1343 training images, 199 validation images, and 261 testing images. It was annotated by non-experts for training and validation, while professional annotators labeled the test set. The dataset is tailored for practical dust analysis in industrial settings, and it has limitations due to challenging conditions.

The DSS Smoke Segmentation dataset Yuan et al. (2019) contains 73632 images, with 70632 for training and 3000 for testing, categorized into DS01, DS02, and DS03 groups.

For evaluation, we use mean Intersection over Union (mIoU) as a comprehensive segmentation accuracy metric.

During CNN training, two hyper-parameters need to be evaluated. One is global color erasing (GCE) probability $p_{g}$. The results of different $p_{g}$ are shown in Fig. 4. We can see that when the value $p_{g}$ is set $0.00$ to $0.15$, the performance of the model is better than the baseline in Rank-1 and mAP. We would like to supplement that when $p_{g}=0$, it represents the accuracy of the original baseline. When $p_{g}=1$, it indicates that the model is exclusively trained with grayscale images. The result of $p_{g}=1$ closely approximates $p_{g}=0.7$. It indicates that training the model only with grayscale images does not yield satisfactory results.

GCE can be viewed as a specific case of random color erasing (RCE). Based on the experimental observations mentioned above, we implement GCE with a probability of $p_{g}=0.15$ within the framework of RCE. When evaluating tatal RCE probability $p_{r}$, we fixed this parameter. The results of different $p_{r}$ are shown in Fig. 4. Obviously, when $p_{r}=0.40$, the model achieves the best performance.

The results of the ablation experiments are also reflected in Fig. 4. It shows that our GCE outperforms the baseline within a probability range of 0.15. Combining local and global color erasing, referred to as RCE, further enhances the baseline. With RCE, we achieve a 1.5 percentage point improvement in Rank-1 and 3.3 percentage point improvement in mAP over the baseline, and a 2.0 percentage point improvement in Rank-1 and 2.7 percentage point improvement in mAP.

We extend our evaluation to State-of-the-Art ReID baselines. As demonstrated in Tab. 3, our method exhibits improvements of 0.6 and 1.3 percentage points in Rank-1 accuracy and mean average precision (mAP) for SB Luo et al. (2019) on Market1501. Moreover, we observe enhancements of 0.2 percentage points in Rank-1 accuracy and 0.9 percentage points in mAP for FastReID He et al. (2023). Importantly, these improvements hold consistently across both DukeMTMC and MSMT17 datasets (see Tab. 2).

Comparison with State-of-the-Art methods. We compare our method with the State-of-the-Art methods, including HOReID Wang et al. (2020), OfM Zhang et al. (2021), PAT Li et al. (2021), DRL-Net Jia et al. (2022), DCAL Zhu et al. (2022) and CLIP-ReID Li et al. (2023). As shown in Tab. 1, we achieve state-of-the-art performance on FastReID by using proposed learning strategy RCE.

Our method enhances accuracy beyond default configurations using data augmentation in FastReID, which includes AutoAugment Cubuk et al. (2019) and random erasing augmentation (REA) Zhong et al. (2020). Notably, the automated data augmentation strategy, AutoAugment, which employs a series of augmentation techniques to simulate color deviations, does not consistently result in improved model performance.

Another comparison of the performance of our method with State-of-the-Art GAN-Based ReID methods is shown in Tab. 4 and Tab. 5. As can be seen from Tab. 4, our method delivers a performance improvement that far exceeds that of DGNet, the GAN-based method, by more than 2.8 percentage points on mAP. As can be seen from Tab. 5, the generalization ability of the proposed method in M→D cross-domain tests is improved by 1.9 percentage points in the Rank-1 compared with the baseline. The improvement is even more pronounced in the M→D cross-domain tests, which further shows that the proposed method is better than DGNet. Notably, using data generated by DGNet for model training leads to poor cross-domain performance.

Cross-domain tests. Cross-domain ReID aims to adapt model training from a labeled source domain dataset to a target domain dataset. However, achieving higher accuracy does not always translate to better generalization, as highlighted by Luo Luo et al. (2019). To investigate the effectiveness of our method in cross-domain experiments, we compare RCE with other methods for cross-domain tests between Market-1501 and DukeMTMC, as illustrated in Tab. 5.

As shown in Table 5, due to the direct disruption of sample structural information during the model learning process by random erasing augmentation (REA), it results in adverse effects in cross-domain scenarios, consequently undermining the model’s performance. In contrast, our RCE demonstrates more pronounced gains in cross-domain testing through ensemble learning at the image level. This clearly indicates that the proposed method effectively guides the model to learn color invariance. This observation emphasizes the exceptional capability of our method in enhancing model robustness against color variations.

Grad-CAM Selvaraju et al. (2017) employs gradients from the final CNN convolutional layer to visualize neuron importance in the output prediction. It highlights regions influencing model predictions. As shown in Fig. 5, the normally trained model activates irrelevant areas under color deviation, whereas RCE-trained model effectively activates critical regions. This indicates that RCE exhibits significant robustness against color deviations.

Observing Tab.6, it can be noted that on both the dust and smoke semantic segmentation datasets, our method consistently demonstrates significant improvements across different architectures of strong baselines, including Segformer Xie et al. (2021) and DDRNet Pan et al. (2023). Notably, our method achieves improvements of over 7 percentage points on DDRNet in both datasets (DustProj and DS03). In addition, we compare our RCE with two methods, namely Cutout and Co-Mixup in Tab. 7. The experiments demonstrate that our approach achieves superior performance in this task.

Our approach consistently improves the results of various baseline models across different tasks and datasets, demonstrating the effectiveness of the proposed method.