TransMatcher: Deep Image Matching Through Transformers for Generalizable Person Re-identification

The Transformer [32] is a neural network based on attention mechanisms. It has shown great success in the field of natural language processing. Recently, it has also shown promising performance for computer vision tasks, including image classification [12, 20], object detection [2, 50, 20, 34], and image segmentation [20, 34], thus gaining increasing attention in this field. However, existing studies mostly use Transformers for feature representation learning, e.g. for image classification or dense predictions, and the generalizability of Transformers is unknown. At a glance, query-key similarities are computed by dot products in the attention mechanisms of Transformers. Therefore, these models could potentially be useful for image matching. In this work, we further investigate the possibility of applying Transformers for image matching and metric learning given pairs of images, with applications in generalizable person re-identification.

Attention mechanisms are used to gather global information from different locations according to query-key similarities. The vanilla Transformer [32] is composed of an encoder that employs self-attention, and a decoder that further incorporates a cross-attention module. The difference is that the query and key are the same in the self-attention, while they are different in the cross-attention. The Vision Transformer (ViT) [12] applies a pure Transformer encoder for feature learning and image classification. While the Transformer encoder facilitates feature interaction among different locations of the same image, it cannot address the image matching problem being studied in this paper, because it does not enable interaction between different images. In the decoder, however, the cross-attention module does have the ability for cross interaction between query and the encoded memory. For example, in the decoder of the detection Transformer (DETR) [2], learnable query embeddings are designed to decode useful information in the encoded image memory for object localization. However, the query embeddings are independent from the image inputs, and so there is still no interaction between pairs of input images. Motivated by this, how about using actual image queries instead of learnable query embeddings as input to decoders?

Person re-identification is a typical image matching and metric learning problem. In a recent study called QAConv [16], it was shown that explicitly performing image matching between pairs of deep feature maps helps the generalization of the learned model. This inspires us to investigate the capability and generalizability of Transformers for image matching and metric learning between pairs of images. Since training through classification is also a popular strategy for metric learning, we start from a direct application of ViT and the vanilla Transformer with a powerful ResNet [6] backbone for person re-identification. However, this results in poor generalization to different datasets. Then, we consider formulating explicit interactions between query¹¹1Query/gallery in person re-identification and query/key or target/memory in Transformers have very similar concepts originated from information retrieval. We use the same word query here in different contexts. and gallery images in Transformers. Two naive solutions are thus designed. The first one uses a pure Transformer encoder, as in ViT, but concatenates the query and gallery features together as inputs, so as to enable the self-attention module to read both query and gallery content and apply the attention between them. The second design employs the vanilla Transformer, but replaces the learnable query embedding in the decoder by the ready-to-use query feature maps. This way, the query input acts as a real query from the actual retrieval inputs, rather than a learnable query which is more like a prior or a template. Accordingly, the cross-attention module in the decoder is able to gather information across query-key pairs, where the key comes from the encoded memory of gallery images.

While the first solution does not lead to improvement, the second one is successful with notable performance gain. However, compared to the state of the art in generalizable person re-identification, the performance of the second variant is still not satisfactory. We further consider that the attention mechanism in Transformers might be primarily for global feature aggregation, which is not naturally suitable for image matching, though the two naive solutions already enable feature interactions between query and gallery images. Therefore, to improve the effectiveness of image matching, we propose a new simplified decoder, which drops the full attention implementation with the softmax weighting, keeping only the query-key similarity computation. Additionally, inspired from QAConv [16], global max pooling (GMP) is applied, which acts as a hard attention to gather similarity values, instead of a soft attention to weight feature values. This is because, in image matching, we are more interested in matching scores than feature values. Finally, a multilayer perceptron (MLP) head maps the matching result to a similarity score for each query-gallery pair. This way, the simplified decoder is computationally more efficient, while at the same time more effective for image matching.

We call the above design TransMatcher (see Fig. 1), which targets at efficient image matching and metric learning in particular. The contributions of this paper are summarized as follows.

• •

  We investigate the possibility and generalizability of applying Transformers for image matching and metric learning, including direct applications of ViT and the vanilla Transformer, and two solutions adapted specifically for matching images through attention. This furthers our understanding of the capability and limitation of Transformers for image matching.

• •

  According to the above, a new simplified decoder is proposed for efficient image matching, with a focus on similarity computation and mapping.

• •

  With generalizable person re-identification experiments, the proposed TransMatcher is shown to achieve state-of-the-art performance on several popular datasets, with up to 6.1% and 5.7% performance gains in Rank-1 and mAP, respectively.

Given pairs of input images, deep feature matching has been shown to be effective for person re-identification. Li et al. [14] proposed a novel filter pairing neural network (FPNN) to handle misalignment and occlusions in person re-identification. Ahmed et al. [1] proposed a local neighborhood matching layer to match deep feature maps of query and gallery images. Suh et al. [28] proposed a deep neural network to learn part-aligned bilinear representations for person re-identification. Shen et al. [25] proposed a Kronecker-product matching (KPM) module for matching person images in a softly aligned way. Liao and Shao [16] proposed the query adaptive convolution (QAConv) for explicit deep feature matching, which is proved to be effective for generalizable person re-identification. They further proposed a graph sampler (GS) for efficient deep metric learning [17].

Generalizable person re-identification has gained increasing attention in recent years. Zhou et al. [49] proposed the OSNet, and showed that this new backbone network has advantages in generalization. Jia et al. [10] applied IBN-Net-b [22] together with a feature normalization to alleviate both style and content variance across datasets to improve generalizability. Song et al. [27] proposed a domain-invariant mapping network (DIMN) and further introduced a meta-learning pipeline for effective training and generalization. Qian et al. [24] proposed a deep architecture with leader-based multi-scale attention (MuDeep), with improved generalization of the learned models. Yuan et al. [42] proposed an adversarial domain-invariant feature learning network (ADIN) to separate identity-related features from challenging variations. Jin et al.[11] proposed a style normalization and restitution module, which shows good generalizability for person re-identification. Zhuang et al. [51] proposed a camera-based batch normalization (CBN) method for domain-invariant representation learning, which utilizes unlabeled target data to adapt the BN layer in a quick and unsupervised way. Wang et al. [36] created a large-scale synthetic person dataset called RandPerson, and showed that models learned from synthesized data generalize well to real-world datasets. However, current methods are still far from satisfactory in generalization for practical person re-identification.

There are a number of attentional networks [18, 23, 19, 41, 26, 15, 40, 43, 8] proposed for person re-identification, but focusing on representation learning. More recently, Zhao et al. [44] proposed a cross-attention network for person re-identificaiton. However, it is still applied for feature refinement, instead of explicit image matching between gallery and probe images studied in this paper.

Transformers have recently received increasing attention for computer vision tasks, including image classification [12, 20], object detection [2, 50, 20, 34], image segmentation [20, 34], and so on. For example, ViT was proposed in [12], showing that a pure Transformer-based architecture is capable of effective image classification. DETR was proposed in [2], providing a successful end-to-end Transformer solution for object detection. Later, several studies, such as the Deformable DETR [50], Swin [20], and PVT [34], improved the computation of Visual Transformers and further boosted their performance. However, existing studies mostly use Transformers for feature representation learning, e.g. for image classification and dense predictions. There lacks a comprehensive study on whether Transformers are effective for image matching and metric learning and how its capability is in generalizing to unknown domains.

For the vanilla Transformer [32], the core module is the multi-head attention (MHA). First, a scaled dot-product attention is defined as follows:

$$\text{Attention}(Q,K,V)=\text{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}})V,$$

(1)

where $Q\in\mathbb{R}^{T\times d_{k}}$ is the query (or target) matrix, $K\in\mathbb{R}^{M\times d_{k}}$ is the key (or memory) matrix, $V\in\mathbb{R}^{M\times d_{v}}$ is the value matrix, $T$ and $M$ are the sequence lengths of the query and key, respectively, $d_{k}$ is the feature dimension of the query and key, and $d_{v}$ is the feature dimension of $V$. In visual tasks, $Q$ and $K$ are usually reshaped query and key feature maps, with $T=M=hw$, where $h$ and $w$ are the height and width of the query and key feature maps, respectively. Then, the MHA is defined as:

$$\text{head}_{i}=\text{Attention}(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}),$$

(2)

$$\text{MultiHead}(Q,K,V)=\text{Concat}(\text{head}_{1},\ldots,\text{head}_{H})W^{O},$$

(3)

where $W_{i}^{Q}\in\mathbb{R}^{d\times d_{k}}$, $W_{i}^{K}\in\mathbb{R}^{d\times d_{k}}$, $W_{i}^{V}\in\mathbb{R}^{d\times d_{v}}$, and $W^{O}\in\mathbb{R}^{hd_{v}\times d}$ are parameter matrices, and $H$ is the number of heads. Then, $Q=K=V$ in the multi-head self-attention (MHSA) in the encoders, while they are defined separately in the multi-head cross-attention (MHCA) in the decoders.

The structure of the Transformer encoder without positional encoding is shown on the left of Fig. 1. Beyond MHSA, it further appends a feed-forward layer to first increase the feature dimension from $d$ to $D$, and then recover it back from $D$ to $d$. Besides, the encoder can be self-stacked $N$ times, where $N$ is the total number of encoder layers. In ViT [12], only Transformer encoders are used, and positional encoding is further applied. In the vanilla Transformer [32], decoders with MHCA are further applied, with the query being learnable query embeddings initially, and the output of the previous decoder layer later on, and the key and value being the output of the encoder layer. The decoder can also be self-stacked $N$ times.

While the above ViT and vanilla Transformer are able to perform image matching through black-box feature extraction and distance learning, they are not optimal for this task because they lack image-to-image interaction in their designs. Though cross-attention is employed in the Transformer decoders, in its original form the query either comes from learnable query embeddings, or from the output of the previous decoder layer.

Therefore, we adapt Transformers with two naive solutions for image matching and metric learning. Building upon a powerful ResNet [6] backbone, the first solution appends ViT, but not simply for feature extraction. Instead, a pair of query and gallery feature maps are concatenated to double the sequence length, forming a single sample for the input of ViT. Thus, both the query and key for the self-attention layer contain query image information in one half and gallery image information in the other half. Therefore, the attention computation in Eq. (1) is able to interact query and gallery inputs for image matching. This variant is denoted as Transformer-Cat.

The second solution appends the vanilla Transformer, but instead of learnable query embeddings, ResNet query features are directly input into the first decoder. This way, the cross-attention layer in the decoders is able to interact the query and gallery samples being matched. This variant is denoted as Transformer-Cross.

The structure of these two variants can be found in the Appendix. Note that these two solutions have high computational and memory costs, especially for large $d$, $D$, and $N$ (c.f. Section 6.4).

Though the above two solutions enable query-gallery interaction in the attention mechanism for image matching, they are not adequate for distance metric learning. This is because, taking a deeper look at Eq. (1) for the attention, it can be observed that, though similarity values between $Q$ and $K$ are computed, they are only used for softmax-based weighting to aggregate features from $V$. Therefore, the output of the attention is always a weighted version of $V$ (or $K$), and thus cross-matching between a pair of inputs is not directly formulated.

To address this, we propose a simplified decoder, which is explicitly formulated towards similarity computation. The structure of this decoder is shown in the middle of Fig. 1. First, both gallery and query images are independently encoded by $N$ sequential Transformer encoders after a backbone network, as shown on the left of Fig. 1. This encoding helps aggregating global information from similar body parts for the subsequent matching step. The resulting feature encodings are denoted by $Q_{n}\in\mathbb{R}^{hw\times d}$ and $K_{n}\in\mathbb{R}^{hw\times d}$, $n=1,\ldots,N$, for the query and gallery, respectively. Then, as in Eq. (2), both the gallery and query encodings are transformed by a fully connected (FC) layer FC₁:

$$Q^{\prime}_{n}=Q_{n}W_{n},K^{\prime}_{n}=K_{n}W_{n},$$

(4)

where $W_{n}\in\mathbb{R}^{d\times d}$ is the parameter matrix for encoder-decoder layer $n$. Different from Eq. (2), we use shared FC parameters for both query and gallery, because they are exchangeable in the image matching task, and the similarity metric needs to be symmetrically defined. Then, the dot product is computed between the transformed features, as in Eq. (1):

$$S_{n}=Q^{\prime}_{n}{K^{\prime}_{n}}^{T},$$

(5)

where $S_{n}\in\mathbb{R}^{hw\times hw}$ are the similarity scores. In addition, a learnable prior score embedding $R\in\mathbb{R}^{hw\times hw}$ is designed, which defines prior matching scores between different locations of query and gallery images. Then, it is used to weight the similarity values:

$$S^{\prime}_{n}=S_{n}*\sigma(R),$$

(6)

where $*$ denotes element-wise multiplication, and $\sigma$ is the sigmoid function to map the prior score embedding into weights in $[0,1]$.

After that, a GMP layer is applied along the last dimension of $hw$ elements:

$$S{{}^{\prime\prime}}_{n}=\text{max}(S^{\prime}_{n},\text{dim=-1}).$$

(7)

This way, the optimal local matching over all key locations is obtained, as in QAConv [16]. Compared to Eq. (1), the GMP here can be considered as a hard attention, but it is used for similarity matching rather than softmax-based feature weighting like in the soft attention. Note that multi-head design in MHA is not considered here (c.f. Section 6.6).

Then, after a batch normalization layer BN₁, an MLP head is further appended, similar to the feed-forward layer of Transformers. It is composed of MLPHead₁=(FC₂, BN₂, ReLU) to map the $hw$ similarity values to dimension $D$, and MLPHead₂=(FC₃, BN₃) to map dimension $D$ to 1 as a single output score $S{{}^{\prime\prime\prime}}_{n}$.

Finally, decoder $n$ outputs a similarity score by fusing the output of the previous decoder:

$$S{{}^{\prime\prime\prime\prime}}_{n}=S{{}^{\prime\prime\prime}}_{n}+S{{}^{\prime\prime\prime\prime}}_{n-1},$$

(8)

where $S{{}^{\prime\prime\prime\prime}}_{0}$ is defined as 0. With $N$ stacked encoder-decoder blocks, as shown in Fig. 1, this can be considered as residual similarity learning. Note that the stack of encoder-decoder blocks in TransMatcher is different from that in the vanilla Transformer. In TransMatcher, the encoder and decoder are connected before being stacked, while in the vanilla Transformer they are stacked independently before connection. This way, the decoder of TransMatcher is able to perform cross matching with different levels of encoded features for residual similarity learning.

However, the GMP operation in Eq. (7) is not symmetric. To make TransMatcher symmetric for the query and gallery, the GMP operation in Eq. (7) can also be applied along dim=0; that is, conduct an inverse search of best matches over all query locations. Keeping other operations the same, this will result in another set of similarity scores, which are summed with the original ones after the FC₃ layer. Further details can be found in the Appendix. Note that this is not reflected in Fig. 1 for simplicity of illustration.

Finally, the outputs of TransMatcher scores for all query-gallery pairs in a batch are collected for pairwise metric learning following the same pipeline in QAConv-GS [17], and the same binary cross entropy loss is used as in the QAConv-GS.

With the study conducted in this paper, we conclude that: (1) direct applications of ViT and the vanilla Transformer are not effective for image matching and metric learning, because they lack cross-image interaction in their designs; (2) designing query-gallery concatenation in ViT does not help, while introducing query-gallery cross-attention in the vanilla Transformer leads to notable but not adequate improvements, probably because the attention mechanism in Transformers might be primarily designed for global feature aggregation, which is not naturally suitable for image matching; and (3) a new simplified decoder thus developed, which employs hard attention to cross-matching similarity scores, is more efficient and effective for image matching and metric learning. With generalizable person re-identification experiments, the proposed TransMatcher is shown to achieve state-of-the-art performance on several popular datasets with large improvements. Therefore, this study proves that Transformers can be effectively adapted for the image matching and metric learning tasks, and so other potentially useful variants will be of future interest.

The authors would like to thank Yanan Wang who helped producing Fig. 1 in this paper, and Anna Hennig who helped proofreading the paper, and all the anonymous reviewers for the valuable feedbacks in improving the paper.