Less is More: Learning from Synthetic Data with Fine-grained Attributes for Person Re-Identification

In the field of person re-ID, early works [45, 23] either concentrate on hand-crafted feature or low-level semantic feature. Unfortunately, these methods always fail to produce competitive results because of their limited discriminative learning ability. Recently, benefited from the advances of deep neural networks, person re-ID performance in supervised learning has been significantly boosted to a new level [36, 5, 22], which learned robust feature extraction and reliable metric learning in an end-to-end manner. Typically, person re-ID model can be trained with the identification loss [41], contrastive loss [34, 35] and triplet loss [17]. Recently, a strong baseline [25] for re-ID is employed to extract the discriminative feature, which has been proved to have great potential to learn a robust and discriminative model in person re-ID models. Besides, several literatures [9, 40] focus on the image-level to allow different domains to have similar feature distributions, or adopt an adversarial domain adaptation approach to mitigate the distribution shift [11, 33], which has attracted considerable attention from various fields in re-ID community.

Being the foundation of more sophisticated re-ID techniques, the pursuit of better datasets never stops in the area of person re-ID. Early attempts could be traced back to VIPeR [15], ETHZ [30] and RAiD [7]. More challenging datasets are proposed subsequently, including Market-1501 [46], CUHK03 [21], MSMT17 [39], etc. However, labelling such a large-scale real-world dataset is labor-intensive and time-consuming, sometimes there even exists security and privacy problems. Besides, all of these datasets only have limited attribute distribution and are lack of diversity. As the performance gain is gradually saturated on the above datasets, newly large-scale datasets are needed urgently to further boost re-ID performance. Recently, leveraging synthetic data is an effective idea to alleviate the reliance on large-scale real-world datasets. This strategy has been applied in various computer vision tasks, e.g., object detection [27], crowd counting [37] and semantic segmentation [6]. In the person re-ID community, many re-ID methods [2, 1, 32, 38] have proposed to take advantage of game engine to construct large-scale synthetic re-ID datasets, which can be used to pre-train or fine-tune CNN network. For example, Barbosa et al. [2] propose a synthetic dataset SOMAset created by photo-realistic human body generation software to enrich the diversity. Recently, Wang et al. [38] collect a virtual dataset RandPerson with 3D characters containing 1,801,816 synthetic images of 8,000 identities. However, these datasets are either in a small scale or lack of diversity, few of them provide rich attribute annotations, which cannot satisfy the need of attribute learning in person re-ID task. So new fine-grained annotated datasets are urgently needed.

Our FineGPR dataset is collected from a popular game engine called the Grand Theft Auto V (GTA5). Practically, we create a synthetic controllable world containing 2,028,600 synthesized person images of 1,150 identities. Images in this dataset generally contain different attributes in a large scope, e.g., Viewpoint, Weather, Illumination, Background and ID-level annotations, also including many hard samples with occlusion. It is worth noting that, all images are simultaneously captured by 36 non-overlapping cameras with a high resolution and image quality. In the process of image generation, each person walks along a schedule route, and cameras are set up and fixed at the chosen locations. As a controllable system, it can satisfy various data requirements in a fine-grained fashion.

The goal of our FineGPR dataset is to introduce a new challenging benchmark with high-quality annotations and multiple attribute distribution to re-ID community. To the best of our knowledge, this is the first large-scale person dataset with over 4 environment-level attributes and 13 ID-level attribute annotations.

Identities. According to the Table 1, FineGPR contains 1,150 hand-crafted identities including females and males, with resolution of $200\times 480$. To ensure diversity, we cropped human region with different angles. As shown in Fig. 1 (second row), different person has different body shape, clothing, hairstyle, and the motion can be randomly set as walking, running, standing and so on. Particularly, the clothes of these characters include jeans, pants, shorts, skirts, T-shirts, dress shirts, etc., and some of these identities have a backpack, shoulder bag, and wear glasses or hat. In total, we manually annotate the FineGPR with 13 different pedestrian attributes at the identity level (e.g., wearing dress or not), the distribution diagram is demonstrated in Fig. 3.

Viewpoint. We construct the image exemplars under specified viewpoints. Those images are randomly sampled during normal walking, running, etc. Formally, a person image is sampled every $10^{\circ}$ from $0^{\circ}$ to $350^{\circ}$. (36 different types of viewpoints in total). There are 49 images for each viewpoint of an identity in the entire FineGPR, so each person has 1,764 ($49\times 36$) images in total.

Weather. Currently, the proposed FineGPR has 7 different weather conditions, including Sunny, Clouds, Overcast, Foggy, Neutral, Blizzard and Snowlight. It is worth mentioning that the number of instances in each weather condition is the same, but not a natural heavy-tail distribution, which makes it adaptable to various real-world scenarios.

Illumination. Illumination is another critical factor that contributes to the success of generalizable re-ID, which consists of 7 different types of illumination, e.g. midnight (time period during 23:00–4:00 in 24 hours a day.), dawn (4:00–6:00), forenoon (6:00–11:00), noon (11:00–13:00), afternoon (13:00–18:00), dusk(18:00–20:00) and night (20:00–23:00). Parameters like time setting can be modified manually for each illumination type. By editing the values of these terms, various kinds of illumination environments can be created.

Background. GTA5 has a very large environment map, including thousands of realistic urban areas and wild scenes. From now, 9 different scenes are selected to represent real-world scenarios with annotations, e.g. street, mall, school, park and mountain, etc., which are distributed evenly across all identities. The different scenes are shown in Fig. 1 (first row). More additional details related to our FineGPR can be found in the Supplementary Material.

Ethical Considerations. People-centric datasets pose some challenges of data privacy [10] and intersectional accuracy disparities [3]. To address such concerns, our dataset were created with careful attention to ethical questions, which we encountered throughout our work. Access to our dataset will be provided for research purposes only and with restrictions on redistribution. Furthermore, we are very cautious of annotation procedure of FineGPR dataset to avoid the social and ethical implications. As for re-ID system, governments and officials must establish strict regulations to control the usage of this technology since it mainly relies on (not all) surveillance data. Motivated by this, we do not consider the dataset for developing non-research systems unless further professional processing or augmentation.

Market-1501 [46] contains 32,668 labeled images of 1,501 identities captured from campus in Tsinghua University. Each identity is captured by at most 6 cameras. The training set contains 12,936 images from 751 identities and the test set contains 19,732 images from 750 identities.

MSMT17 [39] has 126,441 labeled images belonging to 4,101 identities and contains 32,621 training images from 1,041 identities. For the testing set, 11,659 bounding boxes are used as query images and other 82,161 bounding boxes are used as gallery images.

CUHK03 [21] contains 14,097 images of 1,467 identities. Following the CUHK03-NP protocol [48], it is divided into 7,365 images of 767 identities as the training set, and the remaining 6,732 images of 700 identities as the testing set. We adopt mean Average Precision (mAP) and Cumulative Matching Characteristics (CMC) at rank-1 and rank-5 for evaluation on re-ID task.

We mainly use the newly-built FineGPR to conduct the experiments. For attribute optimization, we empirically set $w_{l}=0.2$ in Eq. 2, and $\alpha=0.9$, $\beta=1$ in Eq. 3. It is worth mentioning that our re-ID baseline system is built only with commonly used softmax cross-entropy loss [44] on vanilla ResNet-50 [16] with no bells and whistles. Following the practice in [25], person images are resized to 256 $\times$ 128, then a random horizontal flipping with 0.5 probability is used for data augmentation. The batch size of training samples is set as 128. Adam method [20] is adopted for optimization. The initial learning rate is set to 3.5$\times$10^-4 for the backbone network. Then, these learning rates are decayed to 3.5$\times$10^-5 and 3.5$\times$10^-6 at 40th epoch and 70th epoch respectively, and the training stops after 120 epochs.

To evaluate the superiority of our synthetic dataset, we perform training on FineGPR and testing on each individual real dataset. The evaluation results are reported in Table 2. Surprisingly, when initializing with whole FineGPR dataset, we can achieve a rank-1 accuracy of 50.5%, 12.5% and 8.7% when tested on Market-1501, MSMT17 and CUHK03 respectively. Although there is a slight inferiority of performance when compared with RandPerson [38], our FineGPR selected by AOST with fine-grained attributes can lead a significant improvement by +0.7% and +0.8% in rank-1 accuracy on Market-1501 and CUHK03 dataset respectively. When compared with real-world datasets, FineGPR also outperforms these benchmarks by an impressively large margin in terms of rank-1 accuracy, leading +0.3% and +13.9% improvement on Market-1501 compared with MSMT17 and CUHK03 separately. However, initializing with whole FineGPR dataset is time-consuming and low-efficient, this motivates the investigation of data selecting techniques that can potentially address this problem.

Important Parameters. In Eq. 3, both $\alpha$ and $\beta$ controls the relative importance of the style and content distance respectively between real and synthetic samples. Since $\mathcal{D}_{\text{total}}$ is a linear combination between $\mathcal{D}_{\text{style}}$ and $\mathcal{D}_{\text{content}}$, we can smoothly regulate the emphasis or adjust the trade-off between the content or the style. As depicted in Fig. 6 (a), we observe that when $\alpha/\beta$ is small, the performances is not optimal because the style representation is way too limited to a very small portion, and thus our AOST could only mine the discriminative information in terms of content representation of the re-ID data. The $\alpha/\beta$ should also not be set too large, otherwise the performances drops dramatically since the model mine too many samples in style representation. Specially, $\alpha/\beta=0.9$ yields the best accuracy.

Evaluation of Attribute Importance. Based on the end-to-end AOST system, we evaluate the impacts of different attributes in a fine-grained manner by the XGBoost in terms of gain [4] (feature importance score). As illustrated in Fig. 6 (b-d), it can be easily observed that Identity accounts for the largest proportion no matter which real dataset is employed for testing, followed by Viewpoint attribute. Typically, this conclusion is in accordance with our prior knowledge on generalizable re-ID problem, that is, using more IDs as training samples is always beneficial to the re-ID system, and viewpoint also plays a key role in recognizing the clothes appearance. More details about the importance related to ID-level attributes is provided in the Supplementary. We hope these analysis about attribute importance will provide useful insights for dataset building and future practical usage to the community.

Effectiveness of Attribute Optimization. We proceed study on dependency by testing whether the Attribute Optimization (AO) matters. According to Table 3, our attribute optimization strategy FineGPR+AO (w/o transfer) can lead a significant improvement in rank-1 of +5.6%, +4.5% and +2.6% on Market-1501, MSMT17 and CUHK03 respectively when compared with random sampling (FineGPR+R). We suspect this is due to samples selected by attribute optimization strategy are much closer to real target domain, and the learned attribute distribution has a higher quality, which have a direct impact on downstream re-ID task. Meanwhile, fast training is our second main advantage since the scale of training set can be largely decreased by attribute optimization, e.g., it costs nearly 20 GPU-days¹¹1All timings use one Nvidia Tesla P100 GPU on a server equipped with a Intel Xeon E5-2690 V4 CPU. when pre-training on entire FineGPR with 2,028,600 images. Fortunately, training time will be considerably reduced by 15$\times$ (20 vs. 1.3 GPU-days) by our proposed AOST without performance degradation, which leads a more efficient deployment to real-world scenarios. Surprisingly, even with fewer samples for training, our approach still yields its competitiveness when compared with existing datasets, e.g. 56.3% vs. 55.6% in rank-1 on Market in Table 2, proving the proverbial less-is-more principle.

To go even further, we also adopt AOST on synthetic datasets to prove the priority of FineGPR. Unfortunately, due to lack of fine-grained attribute annotations, these datasets (e.g. PersonX, Unreal and RandPerson) cannot satisfy the need for AOST in re-ID. We instead randomly sample 124,200 images from these datasets individually, and then perform style transfer. As illustrated in Table 3, it can be easily observed that FineGPR+AOST can perform significantly greater than PersonX+R+ST, Unreal+R+ST and RandPerson+R+ST separately, which successfully proves the applicability of our proposed dataset and approach.

Effectiveness of Style Transfer. Synthetic data engine can generate images and annotations at lower labor costs. However, there exists obvious domain gap between synthetic and real-world scenario, which hinders the further improvement in performance on downstream task. Note that for training efficiency, we instead consider an easier but practical strategy, that is, employing off-the-shelf style transfer model to generate photo-realistic images for further effective training. As shown in Table 3, w/o style transfer by AOST, the rank-1 accuracy drops sharply from 56.3% to 45.5% and the mAP drops from 29.2% to 23.8%. This confirms that mitigating domain gap between synthetic and real dataset is a crucial ingredient to make the performance to an excellent level. For simplicity, the SPGAN is used as the style transfer model in the following experiments.

Qualitative Evaluations. Fig. 7 presents a visual comparison of different transfer methods trained on low-resolution Market and high-resolution MSCO respectively. It can be easily noticed that GANs create artifacts and coarse results (indicated by yellow box) when trained on low-resolution images, which still remains problematic. In comparison, our method with MSCO can successfully address the artifacts and produce most visually pleasant results (indicated by red box) in an even better fashion, which is implicitly beneficial to the downstream re-ID mission.

Quantitative Evaluations. Our qualitative observations above are confirmed by the quantitative evaluations. To be more specific, we adopt Fréchet Inception Distance (FID) [18] to measure the distribution difference between synthetic and real photos. Generally, FID measures how close the distribution of generated images is to the real. As shown in Fig. 8, by adding new regulation terms, e.g. attribute optimization or style transfer, the FID score gradually decreases no matter which dataset is employed for evaluation, suggesting the learned attribute distributions are more and more similar to the real images. Even prior to this point, according to Fig. 9, training a generator with low-resolution images always produce low-quality images (indicated by higher FID score) no matter which style transfer model is employed. Still, SPGAN and PTGAN rank the best, while SPGAN shows a slight quantitative advantage. In all, the introduction of high-resolution MSCO dataset can always improve the adaptability to style changes and mitigate previously mentioned domain gap effectively, even in much more complex scenarios.