Viewpoint-aware Progressive Clustering for Unsupervised Vehicle Re-identification

Most existing deep vehicle re-identification methods follow a supervised setting. Pioneer vehicle Re-ID methods [33, 34, 35] focus on the discriminative feature learning. Lou et al. [34] by mining similar negative samples, the features learned by the model are more robust. He et al. [35] proposed an efficient feature preserving method, which can enhance the perception ability of subtle differences. Some works introduce [36, 37, 38, 5, 6, 7] additional attribute information, such as color or type to improve the discrimination of the deep feature for vehicle Re-ID. Temporal path information is also auxiliary information and has been widely employed [39, 40], to improve the robustness of vehicle Re-ID, especially for the vehicles with a similar appearance from the same manufacture. To handle the viewpoint variation issue in vehicle Re-ID, Zhou et al. [5, 6] and Liu et al. [7] employ GAN to infer multi-view information from a single-view of the input image in either image or feature level, to boost the performance by integrating the input and generated images or features. Chu et al. [41] separate the Re-ID into similar and different viewpoint modes, and learn the respective deep metric for each case. In the case of a known 3D bounding box for the vehicle image, Sochor et al. [42] calculated orientation information through 3D coordinates and added to the feature map to improve performance. Despite the significant progress on vehicle Re-ID, these supervised deep learning-based methods require extensive training data, which is expensive in both time and labor-consuming.

Along with the great achievement on person Re-ID, unsupervised person Re-ID offers more challenges, which has attracted more and more attention recently. Recent advances of unsupervised person Re-ID methods generally fall into two categories. 1) The domain adaption based methods [10, 11, 12, 13, 43, 44], which aims to transfer the knowledge in the labeled source domain to the unlabeled target domain. Although the domain adaption based methods make impressive achievement in unsupervised Re-ID by exploring domain-invariant features, they still require a large amount of label annotation in the source domain. Furthermore, the huge diversity in different domains limits their transferring capabilities. 2) The target-only based methods [17, 18, 45], which fulfill the unsupervised task by dividing the unlabeled samples into different categories based on specific similarity. Lin et al. [17] treat each image as a single category and then gradually reduces the number of categories in subsequent clusters. Lin et al. [45] propose a framework that mines the similarity as a soft constraint and introduce camera information to encourage similar samples under different cameras to approach.

To the best of our knowledge, there are few works on unsupervised vehicle Re-ID. Peng et al. [14] propose to use a style GAN to generate vehicle pictures in the source domain more like the target domain. They assume that the source domain contains more viewpoints than the target domain for a better generation. Song et al. [15] introduce the theoretical guarantees of unsupervised domain adaptive Re-ID based on and use a self-training scheme to iteratively optimize the unsupervised domain adaptation model. However, it only focuses on unsupervised domain adaptation, not target-only unsupervised learning. Bashir et al. [46] employ clustering and reliable result selection with embedded color information to iteratively fine-tune the cascade network. However, despite the annotation on color information, this method requires a specific number of identities, which is hard to be known in real-life scenarios.

Due to the extreme viewpoint changes in vehicles, there are relatively small inter-class differences between different vehicles. We argue that global comparison in previous unsupervised clustering methods [17, 18, 15] tends to group the different vehicles with the same viewpoint into the same cluster. Therefore, this global comparison scheme cannot guarantee the promising performance for target-only unsupervised vehicle Re-ID without any label supervision in network training. To handle this problem, we propose to introduce a viewpoint prediction model to identify the vehicle’s viewpoint information during the forthcoming clustering.

In specific, we use a viewpoint prediction network to predict the viewpoint of each unlabeled vehicle image $x_{i}$ in training set $\left\{X\mid x_{1},x_{2},...,x_{N}\right\}$. We train our viewpoint prediction model on VeRi-776 [1], which contains all the visible viewpoints of the vehicle. Following the viewpoints annotation in previous work [22], we divide vehicle images into five viewpoints, e.g., $v=\{front,front\_side,side,rear\_side,rear\}$. Furthermore, we have additionally labeled 3000 samples in VeRi-Wild [32] data to fine-tune the model to improve the robustness of the viewpoint prediction. We use the commonly used cross-entropy loss $L_{\eta}$ to optimize the viewpoint classifier $W\left(x_{i}\mid\theta\right)$,

where $y_{v}$ is a one-hot vector of the ground truth of corresponding viewpoint labels.

After the viewpoint prediction, we can obtain viewpoint-aware unlabeled training set $X^{v}=\{x_{1}^{v},x_{2}^{v},...,x_{N}^{v}\}$, and the current training set can be regarded as the clusters divided according to the viewpoint. For example, VeRi-776 [1] falls into five different viewpoint clusters. For each image in $X^{v}$, we assign a unique index-label $y_{ind}=\{{1},{2},{3},...,{N}\}$ to indicate the category of each sample. In order to learn the discriminative feature, one can achieve this objective by directly using triplet loss [47, 26] or cross-entropy loss via classification. However, the learning driven by these losses, which mainly calculate the similarity among each batch, will become inefficient and difficult to converge with the dataset’s scale growth. Herein, we employ the more efficient repelled loss [23, 48, 17], which calculates the feature similarity between the current sample and all the training samples at once.

It is equipped with a key-value structure to store the features of all training samples, and the index-label $y_{ind}$ is stored in the key memory. The $y_{ind}$ will not change during the entire training process. We calculate the feature similarity between the $i$-th image in the $v$-th viewpoint $f_{i}^{v}$ and all the samples,

where $M[i]$ denotes the $i$-th slot of the value memory $M$. $\beta$ is a hyper-parameter to control the softness of the probability distribution over classes, which is set to 0.1 followed by [17]. $N$ indicates the number of clusters. $y_{p}$ is the pseudo label, and we initialize $y_{p}=y_{ind}$. We maximize the distance between samples by assigning each sample to its own slot,

During the back propagation, the feature memory is updated by the formula $M[y_{i}]\leftarrow\tfrac{1}{2}(M[y_{i}]+f_{i}^{v})$. At the recognition stage, $M[y_{i}]$ stores the features of each training sample. At the subsequent progressive clustering stage, the pseudo label $y_{p}$ of each sample will be redistributed according to the clustering results, while each slot stores the features of each cluster.

Without any identification information, we propose a progressive clustering algorithm for unsupervised vehicle Re-ID. It mainly contains three aspects, two-period algorithm to avoid the similarity dilemma caused by the extreme viewpoint changes of vehicles, the $k$-reciprocal encoding to re-metric the distance for more robust clustering, and clustering with noisy sample selection to deal with outliers that are difficult to be clustered in real scenes.

The first Period

. Through the recognition stage, the model learned more recognizable identity features of each image. The features obtained from the training set $F^{*}=\{F_{1},F_{2},F_{v},...,F_{V}\}$,

where $F_{v}$ and $N_{v}$ represent the feature set and the number of samples in the $v$-th viewpoint. We compare the similarity of all features $F_{v}$ belonging to the same viewpoint cluster to obtain the distance matrix $D(F_{m},F_{n}),\ m=n$. $D$ represents the scoring matrix of Euclidean distance $d_{ij}=\|f_{i}-f_{j}\|^{2}$. There is no doubt that the same vehicle with the same viewpoint has the highest similarity and thus tends to be clustered together (assigned to the same pseudo label) with the highest priority. For each different distance matrix in the same viewpoint, we obtain pseudo labels by the prevalent cluster algorithm DBSCAN [25], which can effectively deal with noise points and achieve spatial clusters of arbitrary shapes without information of the number of clusters compared to the conventional k-means [49] clustering.

The Second Period

. In the second period, we compare the distance between different viewpoint clusters. We take the shortest distance between features in two clusters as a measure of the distance between clusters. Considering that we have no idea whether the current sample has positive samples (with the same identity) in other viewpoints, we comprehensively compare the distance between all different viewpoint clusters,

We argue that the higher similarity, the more likely the same identity. Thus adopt a progressive strategy to merge the clusters between different viewpoints gradually. Therefore, We first calculate a rank list $R$,

where $R$ finds the most similar clusters among all different viewpoints. We set a strict distance threshold $\tau$, and merge clusters from different viewpoints only when the distance of the candidates in $R^{*}$ is less than $\tau$, i.e,

Distance metric by $k$-reciprocal encoding

. Clearly, more positive samples in the same-viewpoint cluster in the first period, higher clustering quality at different viewpoints in the second period, which in turn will benefit the performance in the next iteration. Note that the clustering method significantly relies on the distance metric, we propose introducing the widely used $k$-reciprocal encoding [24, 20, 15] as the distance metric for feature comparison. For the sample $x_{i}^{v}$ in $X^{v}$, we record its $k$ nearest neighbors with index-labels $K_{k}(x_{i}^{v})$, for all indexes $ind\in K_{k}(x_{i}^{v})$, if $\left|K_{k}(_{i}^{v})\cap K_{\frac{k}{2}}(x_{ind}^{v})\right|\geqslant\frac{2}{3}\left|K_{\frac{k}{2}}(x_{ind}^{v})\right|$, $x_{i}^{v}$’s mutual $k$ nearest neighbors set $S_{i}\leftarrow\left|K_{k}(x_{i}^{v})\cup K_{\frac{k}{2}}(x_{ind}^{v})\right|$. In this case, all reliable samples similar to $x_{i}^{v}$ are recorded in $S_{i}$. Then distance $d_{ij}$ of the sample pair in the same viewpoint distance matrix, $D(F_{m},F_{n}),\ m=n$ reassigns weight by,

For each image pairs $(x_{i}^{v},x_{j}^{v})$ at the same viewpoint, we get a new distance matrix $D_{J}(F_{m},F_{n}),\ m=n$ for clustering, it can be calculated by,

where $N_{v}$ is the total number of samples in viewpoint $v$.

Clustering with noisy sample selection

. Our viewpoint-aware clustering strategy avoids comparing different viewpoints of vehicles during the first period of clustering, which alleviates the intra-class gap and reduces the difficulty of clustering to a great extent. However, due to the complexity of the real scene, some hard samples are still difficult to cluster and then regarded as noises. The reason is, although DBSCAN [25] can generate clusters for data of any spatial shape, it uses two parameters $eps$ and $minPts$ to define the density conditions that need to be meet when forming clusters in the training set, which tends to cluster the samples with small intra-class gaps and treat the samples with larger intra-class gaps as noises, as shown in Fig. 3. We observe that these noises usually derive from two situations which are shown as $P_{1}$ and $P_{2}$ in Fig. 3. In $P_{1}$, due to occlusion, misalignment of the bounding box, or deviation of the viewpoint prediction, samples with same identity but far from the already formed clusters (the blue cluster as shown in Fig. 3), will be regarded as noise. In $P_{2}$, some samples deriving from the same identity fail to form into the same cluster since they can not meet the density condition due to large intra-class differences.

For $P_{1}$, we expect noises samples can be classified into clusters with the same identity. For $P_{2}$, we expect that noise samples with the same identity can be clustered together to form a new cluster. Specifically, we use set $S_{n}$ to collect all the noise samples. For each member $s_{i}$ in $S_{n}$, we look for the most similar samples with the same viewpoint $p_{i}$, and constitute a set of similar sample pairs, $\{\{s_{1},p_{1}\},\{s_{2},p_{2}\},...,\{s_{n},p_{n}\}\}$, which is descending sorted according to their pairwise similarities. Then we judge which situation the noise belongs to based on $p_{i}$. If $p_{i}$ belongs to a cluster that has already been clustered, it corresponds to the first situation $P_{1}$. Otherwise, $p_{i}$ is a noise sample, it belongs to the second situation $P_{2}$. However, directly merging $\{s_{i},p_{i}\}$ is not reliable caused by some hard negative samples, we take a more reliable approach as follows. Inspired by $k$-reciprocal encoding, if $\{s_{i},p_{i}\}$ belong to the same identity, their neighbor image sets should be similar, which also means that they should be located in each other’s $\tilde{k}$-nearest neighbors. Therefore, we calculate the $\tilde{k}$-nearest neighbor image sets of the same viewpoint as $p_{i}$. If $s_{i}$ appears in $top\_\tilde{k}[p_{i}]$, $s_{i}$ is regarded as a reliable hard positive sample and will be merged with $p_{i}$. Otherwise, the noisy sample $s_{i}$ will be treated as a hard negative sample and divided into a new cluster to further learn its discriminative feature. Formally, we construct:

We merge $s_{i}$ in $H_{p1}$ with the corresponding $p_{i}$, in $H_{p_{2}}$, we form a new cluster $C_{ni}$ for each pair $(s_{i},p_{i})$, and treat each hard negative sample in $H_{n}$ as a single cluster. Note that when $C_{ni}$ be created, for $c_{i}$ in $C_{ni}$, we will dynamically look for $top\_\tilde{k}[c_{i}]$ as candidates and determine whether to merge the candidate with $C_{ni}$ according to the condition of $H_{p_{2}}$, and not merge with other clusters after merging with $C_{ni}$.

We use ResNet50 [52] as the backbone by eliminating the last classification layer. All experiments are implemented on two NVIDIA TITAN Xp GPUs. We initialize our model with pre-trained weights on ImageNet [53]. For the viewpoint prediction network, we set the batch size as 32 and the learning rate as 0.001, with a maximum 20 epochs. If not specified, we use stochastic gradient descent with a momentum of 0.9 and the dropout rate as 0.5 to optimize the model. For the Re-ID feature extraction network, we resize the input images of VeRi-776 [1] and VeRi-Wild [32] as (384,384). The batch size is set to 16. The learning rates at the recognition stage is set to 0.1 and divided by 10 after every 15 epochs, and set to 0.001 in the clustering stage. We only use a random horizontal flip as a data augmentation strategy. Following the protocol in [24] we set $k$ to 20.

We compare our method with the state-of-the-art unsupervised Re-ID methods on VeRi-776 [1] and VeRi-Wild [32] in both target-only and domain adaption scenarios, as shown in TABLE I.

We further use t-SNE [54] to visualize the feature space distribution of our method compared to the three state-of-the-art target-only methods, as shown in Fig. 4. Compared with ours, the distribution between the points is sparser in OIM [23] and Bottom [17], while more points of different colors gathering in AE [50]. our method presents a better feature distribution, which demonstrates that VAPC_TO can successfully cluster more images of vehicles with the same identity and effectively improve the feature representation for unsupervised vehicle Re-ID.

SPGAN [43] considers the style change among different datasets and trains a style conversion model to bridge the style discrepancy between the source domain data and the target domain. However, due to the huge gap between the vehicle datasets in the real scene, e.g., the diverse viewpoints, resolution and illumination, it is challenging to obtain the desired translated image, which is crucial in SPGAN [43], and thus results in poor performance for vehicle Re-ID. ECN [48] joins the source domain for model constraints while using the $k$-nearest neighbor algorithm to mine the same identity in the target domain. The setting of the $k$ value not only has a greater impact on the experimental results, but the most similar top $k$ samples are always at the same viewpoint. UDAP [15] uses source domain data to initialize the model and theoretically analyzes the rules that the model needs to follow when adapting to the target domain from the source domain. It achieves satisfactory results on vehicle Re-ID due to the strengthening of the constraints on the target domain training. The target domain feature extractor has stronger learnability while obtaining the source domain knowledge. However, it relies on global comparison, which may cause more clustering errors, especially on VeRi-Wild [32] dataset presents a much smaller inter-class differences than VeRi-776 [1].

In addition, we evaluate our method in the ”Direct Transfer” fashion by training on the source domain and directly testing on the target domain indicated as (VAPC_DT) in TABLE I. First of all, by leveraging the information in the training data, VAPC_DT generally outperforms VAPC_TO, which verifies the knowledge of the source domain during the training improves the vehicle retrieval ability of the model. The only exception is the rank-1 score on VeRi-776 [1]. The main reason is the huge gap between VehicleID [51] and VeRi-776 [1] datasets, e.g., VeRi-776 has lower resolution and more viewpoints, which results in poor generalization performance. Even though VAPC_DT significantly boosts the mAP score on both VeRi-776 [1] comparing to the target-only fashion (VAPC_TO). Second, VAPC_DT is even significantly superior to the domain adaption methods SPGAN [43] and ECN [48], and comparable to UDAP [15] on mAP, which proves the robustness of our method for unsupervised vehicle Re-ID.

Note that our method in target-only fashion (VAPC_TO) even surpasses most unsupervised domain adaptation methods such as SPGAN [43] and ECN [48], and works comparably to UDAP [15]. This further verifies the promising performance of our method while handling unsupervised vehicle Re-ID especially without prior annotation information or source data.

In this section, we will thoroughly analyze the effectiveness of three critical components in the VAPC framework, including the two-period (tP) clustering strategy based on viewpoint prediction, $k$-reciprocal encoding (kR) and noise selection (NS), as reported in TABLE II.

Clustering quality is a crucial factor in clustering-based methods for vehicle Re-ID. Therefore, we measure the clustering quality via Adjusted Mutual Information (AMI) [55] on our method compared to the state-of-the-art methods. AMI measures the distribution of ground truth and pseudo labels generated by clustering through mutual information. The larger AMI, the closer distribution of the ground truth and pseudo labels, which in turn means the better clustering quality. We compare our method with Bottom [17], k-means [49] and DBSCAN [25] which also allocate pseudo labels during clustering.

As illustrated in Fig. 8, the classic clustering algorithm k-means [49] and DBSCAN [25] work stumblingly in the global comparison fashion. Furthermore, k-means [49] specifies the number of clusters, which makes the change of samples in the cluster relatively stable. However, due to global comparison, a large number of samples with the same viewpoint and different identities appear in the same cluster, which makes model training continue to decline. DBSCAN [25] is sensitive to noise; therefore, a large number of noise samples under various challenges in real scenes deteriorates the clustering quality. Bottom [17] causes the final collapse due to the accumulation of the number of clustering errors each step. Since clustering based on viewpoint division greatly simplifies the clustering task, and the strategy of progressively merges different viewpoints and gradually gathers vehicles of the same identity from different viewpoints, our method continues to improve with training.

Viewpoint prediction is a prerequisite component in our framework, as discussed in 1. To investigate the influence of viewpoint prediction in our method, we have trained a series of classifiers with different accuracy rates for viewpoint prediction. The experimental results are shown in Fig. 9. As expected, the Re-ID accuracy of VAPC_TO decreases as the accuracy of the viewpoint classification classifier decreases. When the accuracy of the viewpoint classifier drops to 0.5, the accuracy of Rank-1 drops from 76.2% to 70%, (-6.2%) on VeRi-776 [1]. Even though our method with only 0.5 viewpoint classifier accuracy still outperforms the most unsupervised algorithms, as shown in TABLE I. We can see that a robust viewpoint classifier can significantly improve the performance of our algorithm. And due to our more reasonable clustering strategy and effective noise processing, we can also perform well on a poor viewpoint classifier.