StRDAN: Synthetic-to-Real Domain Adaptation Network for Vehicle Re-identification

We developed an ANN using real and synthetic vehicle datasets provided for Track 2 of the 2020 AI City Challenge. The real dataset was the CityFlow-reID dataset, a subset of CityFlow made available for the Track 2 challenge comprising 56,277 images for 666 unique vehicles collected from 40 cameras, with 36,935 images from 333 vehicle identities for training, and 18,290 images from the other 333 identities for testing. The remaining 1052 images in the test set were provided as query data.

The synthetic vehicle dataset comprised 192,150 images from 1,362 distinct vehicles created using the VehicleX [29] synthetic dataset generator, forming an augmented training set. The synthetic dataset includes vehicle ID, color, type, and object orientation; whereas the real dataset includes only vehicle ID. Vehicles were distinguished into 12 colors and 11 types, and orientation was represented by rotation angle [0, 360) on the horizontal plane.

We trained and evaluated the proposed StRDAN model using the VeRi real dataset [8] and City Challenge synthetic data to examine validity and robustness for the approach. The Veri dataset contains over 50,000 images for 776 vehicles captured by 20 cameras, with the training set containing 37,781 images for 576 vehicles, and the testing set containing 11,579 images for 200 vehicles. The VeRi dataset includes labels for vehicle color and type.

Figure 2 shows the proposed overall StRDAN architecture. The model comprises a backbone network for feature extraction and multiple fully connected (FC) softmax layers for classification. Input images are batch sampled in equal numbers from the real and synthetic datasets. For a mini-batch, n different vehicle identities are chosen from the real and synthetic datasets, respectively, then m samples are randomly selected from these chosen images. Therefore, each batch contains $2\times n\times m$ images.

The backbone network extracts a highly abstracted feature vector (dim = 2048) from the input image. In principle, any convolution neural network (CNN) designed for image classification can be used as the backbone network, and a variety of CNNs have been employed in previous studies, including VGG-CNN-M1024 [3], MobileNet [9], and ResNet [7], as vehicle Re-ID model backbone. We selectedResNet-50 as the backbone network for StRDAN. Feature maps extracted by the backbone network are flattened and fed into various FC softmax layers to classify vehicle id, real or synthetic, color, type, and orientation. Outputs are then fed into five cross-entropy loss functions and one triplet loss function. StRDAN was end-to-end trained by updating the network parameters to reduce total loss, combining cross-entropy and triplet losses.

Adversarial Domain Adaptation. An annotated dataset is essential for deep neural network supervised learning. However, collecting and manually annotating large datasets is time consuming and expensive. Therefore, the VehicleX approach was introduced in the AI City Challenge to generate automatically labeled data using a graphic simulator and hence overcome the dearth of real data. However, synthetic data has similar but different distributions than real data. Therefore, it is necessary to train the ANN to predict the classification, regardless of the input domain.

We adopted the adversarial domain adaptation approach, where the ANN learns features that are most discriminative for classification on the real domain and simultaneously as indistinguishable as possible between the real and synthetic domains [1] [5]. To implement this approach, we introduced a domain discrimination layer and its associated cross-entropy loss train the network to be indiscriminate to the domains. We also introduced a vehicle-id classification layer and its associated cross-entropy loss along with triplet loss to train the network to better discriminate vehicle identities and shape signatures.

Semi-supervised Learning. In contrast with the real data, synthetic data includes vehicle type, color, and orientation labels, and we use these labels under multitask learning to improve generalization performance for all tasks [31]. Many semi-supervised learning approaches improve learning accuracy by combining a small amount of labeled data with a large amount of unlabeled data during training. Zhai et al.? s work [30] created artificial labels for unlabeled and labeled data and utilize them in training, and this approach inspired us to use joint and disjoint labels between real and synthetic data to improving performance. Joint labels attached to real and synthetic data are vehicle ID and domain (real or synthetic), whereas disjoint labels were attached to only synthetic data and includes vehicle type, color, and orientation. Losses were classified as joint or disjoint losses, associated with joint and disjoint labels, respectively, as shown in Fig. 2. Triplet loss is classified as a joint loss because the vehicle id contributes to distinguishing batch images into anchor, positive, or negative images.

The semi-supervised learning approach considered here has learning objective

where $\mathcal{L}_{joint}$ is the joint loss defined in both real and synthetic domains and $\mathcal{L}_{disjoint}$ is the disjoint loss defined in the synthetic domain. $\theta$ is parameters of the network. In the next section, we will describe the losses in more detail.

Vehicle ID. Cross-entropy following the softmax function is the most commonly employed loss for image classification, and can be represented for vehicle ID classifier, Lid, as follows

where $N$ denotes the number of images in a mini-batch, $C$ is the number of classes, $y_{ij}$ is the $j^{\mathit{th}}$ element of an one hot encoded vector for the ground-truth of the $i^{\mathit{th}}$ sample in a mini-batch, and $\hat{y}_{ij}$ is the $j^{\mathit{th}}$ element of the softmax FC layer for the $i^{\mathit{th}}$ image.

Domain. We adopted the adversarial domain adaptation approach, with real and synthetic domains. A softmax FC layer was added to the backbone network for domain discrimination, with loss function to train the network to be indiscriminate to the domains,

Domain discrimination loss is defined as the negative of binary cross-entropy loss. Since cross-entropy loss trains the network to discriminative between the domains, the negative loss trains the model to be less discriminating. Thus, if a vehicle captured by a camera is drawn by a graphic simulator in the same orientation, features extracted from a synthetic image would be similar to that from a real image since domain dependent features are suppressed. The negative cross-entropy loss function was implemented by a gradient reversal layer [4].

Triplet Loss. In a mini-batch that contains $P$ identities and $Q$ images for each identity, each image (anchor) has $Q-1$ images of the same identity (positives) and $(P-1)\times Q$ images of different identities (negatives). Triplet loss pulls the positive pair ($a$, $p$) together while pushing the negative pair ($a$, $n$) away by some margin. Thus, triplet loss trains the network to minimize the distance between features from the same image classes and simultaneously maximizes distance between features from different image classes. Triplet loss is defined as [8]

where $v_{a,i}$ is the prediction vector for the $a^{th}$ image of the $i^{th}$ identity group, and $m$ is the margin to control the difference between positive and negative pair distances and helps cluster the distribution more densely.

Color, Type, and Orientation. Softmax cross-entropy loss was applied for these three targets. Orientation is continuous and numerical, whereas color and type are categorical and nominal. Therefore, it seems reasonable to use regression to predict orientation, but orientation is a difficult problem for regression due to the wide range of the regression target. Indeed, experiments showed that optimization would not converge for regression. Therefore, we predicted orientation as direct classification into n discrete bins, with softmax cross-entropy loss [33] or [6], dividing the 360 degrees of orientation space into six bins of 60 degrees. Cross-entropy losses for color, type, and orientation were only applied to synthetic images and set to zero for real images. The loss function can be expressed as

where $x$ is one of color, type, and orientation, and $\delta_{i}$ is a mask value that is set to 1 if the $i^{th}$ data in a mini-batch has $x$, and 0 otherwise.

We used rank-K mean Average Precision (mAP), the official AI City Challenge evaluation metric, to evaluate model performances. mAP measures the mean average precision for each query considering only the top K matches, where we chose K = 100. Average precision was computed for each query image from the area under the precision-recall curve, and then the mean of the average precision over all queries was computed.

The chosen backbone network, ResNet-50, was initialized with weights pre-trained on ImageNet [21] to accelerate training. We trained the model end-to-end with an AMSGrad optimizer [20] for 60 epochs. Initial learning rate = 0.0003, reduced by 0.1 after 20 and 40 epochs. Weight decay factor for L2 regulation was set = 0.0005, and batch size = 64. For each mini-batch, two different vehicle-IDs were selected from each of the real and synthetic datasets, and four images with the same ID were sampled. Therefore, 16 different images with four different IDs from real and synthetic datasets were sampled. Input images were resized to 128 x 256 pixel, and we employed horizontal flip and random erasure augmentations. Postprocessing used the re-ranking algorithm [32], to order the distance matrix between features with Jaccard and original output distance.

We trained and evaluated our models using the CityFlow-reID and VeRi real datasets along with synthetic data generated by VehicleX, with selected disjoint losses as shown in Table 1 and Table 2.

Performance on AI City Dataset. The baseline (Case 1) model comprised the backbone network and vehicle ID classifier. The baseline was trained with the real dataset using vehicle-ID cross-entropy and triplet losses. Table 1 shows that the proposed domain adaptation and semi-supervised learning approaches significantly improved model performance compared with the baseline by at least 8.5% (Case 8) up to 12.9% (Case 4). The proposed model exhibited best performance for Case 4, where only vehicle type was considered, whereas Case 8 considered all three labels and exhibited the worst performance.

Performance on VeRi Dataset. Table 2 also shows that domain adaptation and semi-supervised learning approaches with synthetic dataset and additional losses helped improve performance by at least 1.2% (Case 3) up to 3.1% (Case 2). In contrast to the AI City dataset, the best performance was when considering only orientation (Case 2). However, the model could not converge with the AI City dataset. Veri data model performances were much superior than those for AI City data. Table 3 compares the proposed StRDAN approach with other methods. All models were trained using only the VeRi dataset, aside from PAMTRI and StRDAN (R+S). The proposed StRDAN (R+S) model outperformed all other considered methods.

Domain Adaptation and Semi-supervised Learning. The experimental results verified that domain adaptation and semi-supervised learning approaches help extract more important semantic features for vehicle Re-ID. However, further research is required regarding unexpected phenomena as follows.

• •

  The best model was trained with only one loss of three disjoint losses.

• •

  Performance degraded with including more disjoint losses.

• •

  Best performance depended on the real dataset.