Unity is Strength: Unifying Convolutional and Transformeral Features for Better Person Re-Identification

In recent years, person ReID has achieved impressive performances with deep learning. Typically, image-based person ReID approaches focus on extracting discriminative features [13]. Prior to Transformers, CNN-based methods have dominated person ReID. The importance of local features is first suggested in [28]. Typical methods (e.g., PCB [29], MGN [30]) divide person images into several stripes to obtain multi-grained representations. Besides, Yao et al. [31] introduce a part-based loss to extract fine-grained features from different local parts. Auxiliary information such as human pose [32, 33], parsing [34, 35] is also used for discriminative feature extraction. Other than focusing on the local parts of persons, other researchers try to utilize CNNs to extract more contextual features [36, 37]. However, limited by the local receptive field of CNNs, the contextual relationship between the human parts is not fully extracted. Inherently, the attention mechanism in Transformers adaptively models the relationship between all the human parts. Thus, the superiority of Transformers can help us alleviate this problem.

Transformers have also been introduced to person ReID. As a primary work, He et al. [16] present a pure Transformer-based method named TransReID for object re-identification. Zhu et al. [17] adopt an automatic alignment to extract local features for accurate person ReID. Li et al. [38] propose a harmonious attention network to learn robust feature representations. Li et al. [39] introduce an encoder-decoder Transformer architecture for person ReID. Li et al. [40] leverage frequency-aware information for more robust feature extraction. Although these methods show better results, Transformers lack the inductive bias of CNNs. It has a weak perception of details such as textures, which leads to the ambiguity in persons and backgrounds. Fortunately, CNNs are sensitive to low-level information, so it is necessary to introduce the desired properties for person ReID.

Some researchers are focusing on how to effectively fuse the features of CNNs and Transformers. For example, Conformer [41] and Mobile-Former [27] are designed with a parallel structure that allows continuous interaction of deep features. Dai et al. [42] unify depthwise convolutions and self-attentions for visual recognition. Zhang et al. [13] insert Transformer layers into different levels of features in CNNs. Chen et al. [10] enhance the features through channel and spatial attention modules for CNN backbones. For robust person ReID, Wang et al. [15] utilize a CNN backbone and the neighbor cluster method for Transformers. These previous methods [43, 44, 45] typically insert one structure into another or interactively stack two structures. In the case of two-branch structures, the continuous interaction starts at low-level feature extraction, potentially leading to the destruction of deep features. In this work, we abandon the complex structures and use a parallel feature fusion, preserving the inherent properties of CNNs and Transformers. Besides, our method is general and the used backbone can be easily replaced.

To jointly extract deep features, previous works mainly modify the basic structures of CNNs and Transformers. Different from them, we directly utilize the CNNs and Transformers as our backbone networks. As shown in the left part of Fig. 2, the ResNet50 [4] is deployed to extract the convolutional features $\textbf{X}_{c}\in R^{D_{c}\times H_{c}\times W_{c}}$. Here $D_{c}$, $H_{c}$ and $W_{c}$ present the channel number, height and weight, respectively. Meanwhile, the ViT-B/16 [5] is used to extract transformeral features $\textbf{Z}\in R^{D_{t}\times(1+N)}$. Here, $D_{t}$ denotes the embedding dimensions and $N$ represents the number of image patches. Additional learnable class token is also utilized to aggregate the patch features. We also add the camera embedding [16] to all the tokens to enhance the feature discrimination. In the CNN branch, $\textbf{X}_{c}$ is followed by a Generalized Mean Pooling (GeMP) [48] to obtain the final feature vector $\hat{f}_{c}\in R^{D_{c}}$. In the Transformer branch, Z is divided into $\hat{f}_{t}\in R^{D_{t}}$ and $\textbf{X}_{t}\in R^{D_{t}\times N}$. Furthermore, $\hat{f}_{c}$ and $\hat{f}_{t}$ are utilized for loss supervision. $\textbf{X}_{c}$ and $\textbf{X}_{t}$ are passed into DMF for the subsequent feature fusion.

After extracting features by DFE, we propose a novel DMF for deep feature fusion, as depicted in Fig. 2. Although feature concatenation can be utilized, this simple fusion strategy fails to fully exploit local features from CNNs and global features from Transformers. Therefore, our approach adopts the attention mechanism, including self-attention and cross-attention, to achieve a deep mutual fusion between the convolutional and transformeral features. More specifically, our proposed DMF consists of Local Refinement Units (LRU) and Heterogenous Transmission Modules (HTM). LRU is used to achieve feature alignment. HTM includes a Shared Encoding Unit (SEU) and two Mutual Fusion Units (MFU). The stacked HTM enables the reuse of deep features from CNNs and Transformers.

Local Refinement Unit. When adopting different backbones, the dimensions of output features may not be consistent. Thus, we first utilize the LRU to adjust the spatial sizes and channel dimensions of convolutional and transformeral features. Besides, LRU is useful for reducing the computational burden and increasing the efficiency. Meanwhile, for the transformeral features, LRU employs local interactions between addjacent patches, which is beneficial for introducing the spatial information. Specifically, taking the transformeral features as an example, $\textbf{X}_{t}\ \in R^{D_{t}\times N}$ is firstly reshaped to $\hat{\textbf{F}}_{t}\in R^{D_{t}\times H_{t}\times W_{t}}$. Then, the depth-wise convolution and point-wise convolution are utilized by:

where $\delta$ presents the PReLU [49]. $\phi$ and $\varphi$ present the depth-wise convolution and point-wise convolution with a Batch Normalization (BN) layer [50], respectively. The depth-wise convolution is applied to change the spatial size of features, then the point-wise convolution to reduce the channel number. Similarly, the convolutional features $\hat{\textbf{F}}_{c}\in R^{D_{c}\times H_{c}\times W_{c}}$ directly copied from $\textbf{X}_{c}$ is also passed into another LRU. Finally, we obtain the refined features $\textbf{F}_{c}\in R^{D\times H\times W}$ and $\textbf{F}_{t}\in R^{D\times H\times W}$. After applying the Generalized-Mean (GeM) pooling [48] on $\textbf{F}_{c}$ and $\textbf{F}_{t}$, $f_{(c,0)}^{\prime}$ and $f_{(t,0)}^{\prime}$ are utilized to supervise the CNN and Transformer backbones, respectively.

Shared Encoding Unit. In this work, SEU is designed to enable deep feature enhancement. We employ a self-attention mechanism to explore the dependencies among features, highlighting informative regions and suppressing irrelevant noises. As shown in Fig. 3(b), the transformeral features $\textbf{F}_{t}$ obtained from LRU are first flattened. Then it is concatenated with the global token $f_{(t,0)}^{\prime}$ to form $\textbf{F}_{(t,0)}^{\prime}\in R^{D\times(H_{t}W_{t}+1)}$. After that, $\textbf{F}_{(t,0)}^{\prime}$ is passed into SEU. In SEU, the Layer Normalization (LN)[51], Multi-Head Self-Attention (MHSA)[5] and Feed-Forward Network (FFN)[5] with residual connection [52] are utilized to yield transformeral features:

Similarly, the convoluational features $\textbf{F}_{(c,0)}^{\prime}$ is also passed into another SEU to obtain $\textbf{S}_{(c,0)}$ for deep feature enhancement. The SEU is shared in each HTM and encodes the convolutional and transformeral features for MFU.

Mutual Fusion Unit. MFU allows for global and local interactions between features of different attributes, forming a unified fusion framework through symmetric operations on tokenlized features. Specifically, MFU utilizes the cross-attention mechanism to achieve the deep feature fusion between the convolutional and transformeral features. As shown in Fig. 3(c), we first feed the global convolutional feature $\textbf{S}_{(c,0)}^{g}$ and local transformeral features $\textbf{S}_{(t,0)}^{l}$ into MFU. In MFU, we employ a Multi-Head Cross-Attention (MHCA), which consists of $N_{h}$ heads. More specifically, in the $h$-th head of the CNN branch, the global token $\textbf{S}_{(c,0)}^{g}$ is passed into a linear transformation to generate $\textbf{Q}_{(c,0)}$, which is seen as the Query. The local features $\textbf{S}_{(t,0)}^{l}$ from the Transformer branch are passed into two linear transformations to generate $\textbf{K}_{(t,0)}$ and $\textbf{V}_{(t,0)}$, which are seen as the Key and Value, respectively. Here, $\textbf{Q}_{(c,0)}$, $\textbf{K}_{(t,0)}$, $\textbf{V}_{(t,0)}\in R^{d\times H_{t}W_{t}}$, $d=\frac{D}{N_{h}}$. By utilizing them, the cross-attention is defined as:

where $\sigma$ is the softmax activation function and $(\cdot)^{\top}$ means the transposition operation. The outputs of multiple heads ($\hat{\textbf{M}}^{1}_{(c,0)},\cdots,\hat{\textbf{M}}^{h}_{(c,0)},\cdots,\hat{\textbf{M}}^{N_{h}}_{(c,0)}$) are concatenated to be $\hat{\textbf{M}}_{(c,0)}$. In the $h$-th head of MHCA, the global feature $\textbf{S}_{(t,0)}^{g}$ from the Transformer branch is passed into a linear transformation to generate $\textbf{Q}_{(t,0)}$, which is seen as the Query. The local features $\textbf{S}_{(c,0)}^{l}$ from the CNN branch are passed into two linear transformations to generate $\textbf{K}_{(c,0)}$ and $\textbf{V}_{(c,0)}$, which are seen as the Key and Value, respectively. Similarly, $\textbf{Q}_{(t,0)}$, $\textbf{K}_{(c,0)}$, $\textbf{V}_{(c,0)}\in R^{d\times H_{c}W_{c}}$. The interaction is formulated as:

Here, ($\hat{\textbf{M}}^{1}_{(t,0)},\cdots,\hat{\textbf{M}}^{h}_{(t,0)},\cdots,\hat{\textbf{M}}^{N_{h}}_{(t,0)}$) are concatenated to be $\hat{\textbf{M}}_{(t,0)}$. Then, by utilizing different FFNs, the features $\hat{\textbf{M}}_{(c,0)}$ and $\hat{\textbf{M}}_{(t,0)}$ are further encoded to obtain $f_{(c,1)}^{\prime}$ and $f_{(t,1)}^{\prime}$:

Here, each FFN has two linear projections with the Gaussian Error Linear Unit (GELU) [53]. Finally, by utilizing the cross-attention mechanism, our proposed MFU could fuse two types of deep features to generate the unified representations.

Stacking of HTM. In this work, we combine a SEU and two MFUs as the Heterogenous Transmission Module (HTM). As shown in Fig. 2, we continuously stack HTM to fuse the deep features from the two-branch framework. Noted that, $f_{(c,1)}^{\prime}$ will be concatenated with the patch tokens in $F_{(c,0)}^{\prime}$ again, As shown in Fig. 3 (a), in the $k$-1 step, $f_{(c,k-1)}^{\prime}$ has fused the information from the Transformer branch. Then, it is concatenated with initial deep features from $\textbf{F}_{(c,0)}^{\prime}$, following self-attention to model their internal relationships again. By interacting with the local features, the transmission of heterogeneous information is achieved. The symmetry operation will be applied to $f_{(t,k-1)}^{\prime}$. Through this continuously stacking, we unify two types of deep features. Thus, the mutual guidance and heterogenous transmission are achieved. In this way, it enables DMF to use different information at multiple levels. Then, the output $f_{(c,L)}^{\prime}$ and $f_{(t,L)}^{\prime}$ of the last layer are obtained for loss supervision. Finally, as shown in Fig. 2, six features are supervised during training. During testing, we concatenate these six features to form $f_{a}$ for the final retrieval.

As shown in Fig. 2, six output features from DFE and DMF are supervised by the label smoothing cross-entropy loss [46] and triplet loss [47]. More specifically, the features $\hat{f_{c}}$ and $\hat{f_{t}}$ from the CNN branch and Transformer branch in DFE are first supervised for dual-branch learning. Taking $\hat{f_{c}}$ as an example, the label smoothing cross-entropy loss can be defined by:

where $y^{j}$ is the corresponding ground-truth label. $J$ denotes the total number of classes. In a mini-batch, we build a feature triplet set $\Omega=\{\emph{{F}}_{a},\emph{{F}}_{p},\emph{{F}}_{n}\}$. Here, $\emph{{F}}_{a}$ presents the anchor features. $\emph{{F}}_{p}$ and $\emph{{F}}_{n}$ present its positive features and negative features, respectively. Thus, the triplet loss can be defined by:

where $\|\cdot\|_{2}$ refers to the L2 norm. Afterwards, to improve the discriminative ability [54], we also employ the supervision after aligned features ($f_{(c,0)}^{\prime}$ and $f_{(t,0)}^{\prime}$) and fused features ($f_{(c,L)}^{\prime}$ and $f_{(t,L)}^{\prime}$) in DMF. Finally, we optimize the whole framework with six features simultaneously by a total loss:

We conduct experiments on three large-scale ReID datasets, i.e., Market1501 [55], DukeMTMC [5] and MSMT17 [56]. Market1501 consists of 32,668 images of 1,501 identities. DukeMTMC consists of 16,522 training images of 1,404 identities. It has 2,228 queries images and 17,661 gallery images. MSMT17 is the most challenging dataset. There are 126,441 images of 4,101 identities observed under 15 different cameras. All images provide their camera IDs in three datasets, which can be used as additional information when training and testing. Following previous works, we adopt the mean Average Precision (mAP) and Cumulative Matching Characteristics (CMC) at Rank-1 as our evaluation metrics.

In this work, we realize our model with the Pytorch toolbox, and conduct experiments on two NVIDIA A800 GPUs. In DFE, we employ the pre-trained CNNs and Transformers on the ImageNet dataset [57] as our backbones. All person images are resized to 256$\times$128. When training, these images are augmented by random horizontal flipping, random cropping and random erasing [58]. The min-batch size is set to 128, containing 8 identities and 16 images per identity. During training, we adopt the Stochastic Gradient Descent (SGD) optimizer with a momentum of 0.9 and the weight decay of 1e^-4. Besides, the learning rate is initialized as 5e^-4. In the first 10 epoch, we adopt a warmup strategy with a linearly growing learning rate to 5e^-3. Then, it decreases with a cosine decay for the next 170 epochs.

Tab. I shows the evaluation on three public ReID benchmarks, i.e., Market1501, DukeMTMC and MSMT17.

Market1501: On Market1501, we can see that our method achieves the best mAP and Rank-1. Noted that, RGA-SC [14] gains a comparable Rank-1 on Market1501, which learns a global attention to refine the features. Different from RGA-SC, we utilize HTM to encode and fuse two types of deep features. Thus, our method has a higher mAP than RGA-SC. The best performances show that our proposed framework is able to unify the strengths of CNNs and Transformers to get more discriminative representations.

DukeMTMC: Our method achieves the best Rank-1 and mAP on DukeMTMC. Noted that, NFormer [15] utilizes Transformers to model the relationships between all images, obtaining robust representations. Different from NFormer, we intergrate the heterogeneous information from CNNs and Transformers, resulting in a higher Rank-1 and mAP.

MSMT17: As shown in Tab. I, our method achieves the best mAP and Rank-1 on MSMT17. It is noteworthy that although NFormer performs well on small datasets, it performs poorly on the most challenging large-scale dataset, MSMT17. Although it has a lower number of parameters and FLOPs, our method outperforms it by about 10% on mAP and Rank-1. The results also demonstrate the stronger generalization performance of our method on different scale datasets. Overall, our method attains better performances on three public ReID benchmarks than most state-of-the-art methods. The best mAP and Rank-1 demonstrate the superiors of our method even with low image resolutions, especially on the large-scale dataset.

Performances with Different Configurations: In the last four rows of Tab. I, we compare the performance of FusionReID with different configurations, including the input image resolution and the overlapping of ViT. Generally, using high-resolution images and patch overlapping in ViT can improve the performance of FusioReID on all three datasets. High-resolution images contain more visual information, and the patch overlapping enhances correlations between neighboring areas. These superior performances demonstrate the generalization ability of our proposed FusionReID.

Effectiveness of Key Components. We conduct experiments to verify the effectiveness of proposed key components. The ablation results are shown in Tab. II. Comparing the results of the first three rows, one can see that although DFE only concatenates two types of backbone features, higher ReID accuracies are achieved. Furthermore, we employ our DMF after DFE. The results are reported in the last four rows of Tab. I. When utilizing LRU, the performance has been slightly improved. After feature alignment in LRU, we further utilize SEU or MFU to encode and fuse deep features. Therefore, compared with DFE, the utilization of SEU gains 1.0% mAP and 0.7% Rank-1 improvements, and the utilization of MFU gains 1.2% mAP and 0.6% Rank-1 improvements. Furthermore, our method combines all proposed key components and achieves the best performance, i.e., 69.5% mAP and 86.7% Rank-1. Overall, the gradual increase in performances validates the effectiveness of our proposed key components.

Effects of Various Structures in HTM. As shown in Fig. 4, we design various structures in HTM. The ablation results with various structures are shown in Tab. III. Here, the computation cost only considers the HTM part. One can find that the best performance is achieved by SEU first and then MFUs. The reasons may be twofolds: (1) The shared SEU in front can constrain the distributions of different deep features, which is helpful for the subsequent fusion. (2) The unshared MFU in back can utilize deep mutual fusion to generate better features, which highlights the importance of feature fusion in the last step. Among the four structures, the worst performance is achieved by MFU first and then SEU. There are two reasons for the poor results: (1) For the first HTM layer, the differences of deep features are significant, and it is difficult to fully integrate them through MFU. (2) For the subsequent HTMs, they only change the weighted parameters of global features. When it goes directly to the MFU of next HTM, it is still fusing with each other’s initial deep features. The difference problem in feature distribution still exists. Therefore, the order of SEU and MFU is very important. Thus, we deploy SEU first and then MFU to achieve better deep fusions.

Effects of Weight-Sharing in SEU and MFU. In Tab. IV, we analyze the effect of weight-sharing in SEU and MFU. The weight-sharing in SEU reduces feature differences across branches, improving MFU fusion. Conversely, unshared MFUs allow each branch to focus on distinct fusion patterns, enhancing their effectiveness. The combination of weight-sharing SEU and unshared MFU delivers the best overall performance.

Effects of Different Pooling Structures in DMF. In Tab. V, we compare the performance of different pooling structures in DMF. The GeMP clearly outperforms the Global Average Pooling (GAP), demonstrating its effectiveness to capture the distinctiveness with negligible computational cost.

Complementarity between CNNs and Transformers. We employ different backbone networks in DFE to verify the complementarity between CNNs and Transformers. In the CNN branch, we choose residual networks [4], including ResNet50/101/152. In the Transformer branch, we choose T2T-ViT-14 [65], ViT-B/16 [5] and DeiT-B/16 [66]. As shown in Tab. VI, we find that the performance of ViT-B/16^∗ and DeiT-B/16^∗ outperforms ResNet50/101/152. And for T2T-ViT-14, its corresponding features are also improved considerably with the help of CNNs. When unifying ResNet50 and ViT-B/16^∗, the fused feature $f_{(c,L)}^{\prime}$ attains 66.0% mAP and 85.4% Rank-1, which has 11.7% mAP and 7.3% Rank-1 improvement over ResNet50. Besides, the fused feature $f_{(t,L)}^{\prime}$ attains 3.2% mAP and 1.6% Rank-1 improvement over the results of ViT-B/16^∗, respectively. Moreover, the concatenated feature $f_{a}$ has a better mAP and Rank-1. These improvements suggest that our proposed framework has a significant effectiveness on unifying convolutional and transformeral features. In the same way, when combining ResNet50/101/152 and DeiT-B/16^∗, all the metrics are increased significantly. In particular, when unifying ResNet152 and ViT-B/16^∗, the overall feature $f_{a}$ obtains 71.8% mAP and 87.9% Rank-1. The results clearly demonstrate the complementarity between CNNs and Transformers. Meanwhile, they illustrates the generalization of our proposed method with different backbone networks.

Influence of the Stacked HTM. We conduct experiments on MSMT17 to verify the influence of the stacked HTM. The results are shown in Fig. 5. One can see that the accuracy curve changes but not significantly. As the number of stacked layers increases, the model performance gradually improves, especially from 1 to 2 layers. The refined features (middle curves) after LRU are stronger than the original features (lower curves), while the fused features (upper curves) are better than the refined features after LRU. The improvement indicates that the stacked HTM could better extract discriminative features. However, the performance is saturated when stacked layers are more than 2. Thus, we set the layers to 2 by default.

Influence of Fused Dimensions in DMF. Fig. 6 shows the influence of using different fused dimensions. We change the feature dimensions from 384 to 1536. We find that the performance is worst when the dimension is 384, and the performance is best when the dimension is 1152, achieving 70.0% mAP and 86.8% Rank-1. With the dimensions keep rising, the performance of mAP and Rank-1 begins to decrease. The reason may be that the model tends to over-fitting when the dimension is too high. Although the fused dimension of 384 has the smallest number of model parameters and FLOPs, it outperforms most sate-of-the-art methods. To avoid excessive model complexity, we set 768 as the default dimension.

Analysis of Computational Cost. The ablation analysis of computational cost is also presented. In Tab. II, as the accuracy increases, the computational cost increases. In fact, the computational complexity of our proposed method is acceptable. As shown in Fig. 6, when we reduce the fused dimension to 384, the slight increase in model memory and computational complexity results in considerable accuracy gains. For example, in comparison to TransReID, our approach demonstrates superior performance across three datasets, even with a moderate increase in parameters and FLOPs. This also proves that the effectiveness of our method does not come from simple model stacking and parameter increasing.

In this subsection, we further explain the effectiveness of our approach through visualization analysis. First, we compare feature maps from typical CNNs and Transformers to verify their complementarities. Besides, we visualize attention maps in our DMF to illustrate the effects of deep fusions.

Visualization of Features in LRU and HTM. As shown in Fig. 7, we utilize Grad-CAM [67] to visualize the feature maps in LRU and HTM. Fig. 7(a) shows the original images of six persons. Comparing Fig. 7(b) and Fig. 7(g), one can see that convolutional features generally focus on meaningful local regions of persons, while transformeral features receives the global perception of persons. The differences verify their unique strengthes between CNNs and Transformers. Notably, Fig. 7(g) and Fig. 7(h) demonstrate that Transformers imitate the local perception of CNNs. More edge information is captured by the transformeral features. Comparing Fig. 7(c), Fig. 7(d), Fig. 7(h) and Fig. 7(i), one can see that the convolutional features after LRU appropriately highlight the regions of interest. Different from Fig. 7(d), Fig. 7(e) shows the convolutional features after HTM begin to focus on more regions for comprehensive perceptions. A similar phenomenon occurs in Fig. 7(i) and Fig. 7(j). Noted that, the highlighted regions from CNNs and Transformers are complementary. These visualizations provide a more intuitive understanding of the mutual fusion in our proposed methods.

Visualization of Different Heads in HTM. In Fig. 8, we visualize the attention weights of different heads in HTM, using the same example as in Fig. 9. As shown in Fig. 8(a), different heads focus on different areas. After the first MFU layer (Fig. 8(b)), the Transformer branch shifts to key parts of the body, with a smoother weight distribution, similar to CNN. After SEU (Fig. 8(c)), attention regions change, and the second MFU layer (Fig. 8(d)) narrows the focus to finer details like the head, legs, and briefcase. Meanwhile, after the second MFU layer (Fig. 8(h)), the CNN branch captures the entire human body with discriminative information. These visualizations confirm the effectiveness of the mutual guidance in our proposed methods.

Visualization of Attention Weights in SEU and MFU. As shown in Fig. 9, we visualize the attention weights in SEU and MFU. From the results, one can see that SEU can enhance their unique characteristics in the CNN branch and Transformer branch. For example, in the CNN branch, SEU merely assigns high attention weights to the meaningful local regions. In the Transformer branch, highlighted regions cover the human body and background with a global perception. Meanwhile, Fig. 9(c) and Fig. 9(e) clearly show that the MFU fuses their unique characteristics, which are different with Fig. 9(b) and Fig. 9(d). These visualizations could explain the mechanisms of feature enhancement and mutual fusion in SEU and MFU. Moreover, the changes of attention weights in SEU and MFU validate the effectiveness of our DMF.