Supplementary Material: Channel Recurrent Attention Networks for Video Pedestrian Retrieval

In the channel recurrent attention module, we use an LSTM to jointly capture spatial and channel information. In the implementation, we feed the spatial vectors to the LSTM sequentially, such that the recurrent operation of the LSTM captures the channel pattern while the FC layer in the LSTM has a global receptive field of each spatial slice. Since the LSTM is a temporal model and its output depends on the order of the input sequences, we analyze how the order of the input spatial vectors affects the attention performance and whether the LSTM has the capacity to learn the pattern in the channel dimension of the feature maps.

In the main paper, we feed the spatial vectors to the LSTM sequentially in a \sayforward manner (i.e., from $\hat{{x}}_{1}$ to $\hat{{x}}_{\frac{c}{d}}$). We continue to define other configurations to verify how the sequence order affects the attention performance. The \sayreverse configuration: feeding the spatial vectors from $\hat{{x}}_{\frac{c}{d}}$ to $\hat{{x}}_{1}$ to the LSTM, whose direction is opposite to that in the \sayforward configuration. In the \sayrandom shuffle configuration, we first randomly shuffle the order of spatial vectors in $\hat{{\boldsymbol{x}}}$, and then feed them to the LSTM sequentially. Then we recover the produced $\hat{{\boldsymbol{h}}}$ and generate the attention maps. This \sayrandom shuffle is operated in each iteration during training. The last configuration, we term \sayfixed permutation. In this configuration, we randomly generate a permutation matrix (i.e., ${\boldsymbol{p}}_{1}$) and apply to $\hat{{\boldsymbol{x}}}$, to produce $\hat{{\boldsymbol{x}}}^{p}$ (i.e., $\hat{{\boldsymbol{x}}}^{p}={\boldsymbol{p}}_{1}\hat{{\boldsymbol{x}}}$). Then we feed each row of $\hat{{\boldsymbol{x}}}^{p}$ to the LSTM and obtain $\hat{{\boldsymbol{h}}}^{p}$ and apply another permutation matrix ${\boldsymbol{p}}_{2}$, as $\hat{{\boldsymbol{h}}}={\boldsymbol{p}}_{2}\hat{{\boldsymbol{h}}}^{p}$. Here, ${\boldsymbol{p}}_{2}={\boldsymbol{p}}_{1}^{\top}$ and ${\boldsymbol{p}}_{1}$, ${\boldsymbol{p}}_{2}$ are fixed during training. For this configuration, we perform the experiments twice with two different permutation matrices.

We empirically compare the aforementioned four configurations on the iLIDS-VID and the MARS datasets, shown in Table 1. From Table 1, we observe that the LSTM does indeed learn useful information along the channel dimension via the recurrent operation (i.e., row (ii), (iv), (v) and (vi)) when the order of the spatial vectors is fixed during training. However, if we randomly shuffle the order of the spatial vectors before feeding to them to the LSTM in each iteration (i.e., row (iii)), the LSTM fails to capture useful information in the channel, and the attention mechanism even degrades below the performance of the baseline network on the MARS dataset (i.e., row (i)).

In this analysis, we can draw the conclusion that the order of the spatial vectors has a minor influence on the attention performance when the order of spatial vectors is fixed. However, it is still difficult to figure out the optimal order of spatial vectors. In all experiments, we empirically use the \sayforward configuration in our attention mechanism.

Intuitively, we further use Bi-LSTM to replace the LSTM in the channel recurrent attention module, to verify whether the sophisticated recurrent network is able to learn more complex information in the channel dimension. Table 2 compares the difference of LSTM and Bi-LSTM in channel recurrent attention module. This study shows that the attention w/ Bi-LSTM cannot brings more performance gain than the that w/ LSTM. However, the Bi-LSTM almost doubles the computation complexities and parameters. Thus we choose regular LSTM in our attention module.

In this section, we show the analysis of modeling the video clip as a set and the set aggregation cell acting as a valid set function.

In our channel recurrent attention network, we sample $t$ frames in a video sequence randomly, to construct a video clip (i.e., $[T^{1},\ldots,T^{t}],T^{j}\in\mathbb{R}^{C\times H\times W}$) with its person identity as label (i.e., $y$). In such a video clip, the frames are order-less and the order of frames does not affect the identity prediction by the network during training. The video frames are fed to the deep network and encoded to a set of frame feature vectors (i.e., ${\boldsymbol{F}}=[{\boldsymbol{f}}^{1},\ldots,{\boldsymbol{f}}^{t}],{\boldsymbol{f}}^{j}\in\mathbb{R}^{c}$), then the frame features are fused to a discriminative clip representation (i.e., ${\boldsymbol{g}}$) by the aggregation layer (i.e., set aggregation cell).

The set aggregation cell realizes a permutation invariant mapping, $g_{\kappa}:\mathcal{F}\to\mathcal{G}$ from a set of vector spaces onto a vector space, such that the frame features (i.e., ${\boldsymbol{F}}=[{\boldsymbol{f}}^{1},\ldots,{\boldsymbol{f}}^{t}],{\boldsymbol{f}}^{j}\in\mathbb{R}^{c}$) are fused to a compact clip representation (e.g., ${\boldsymbol{g}}\in\mathbb{R}^{c}$). If in this permutation invariant function (i.e., $g_{\kappa}$), the input is a set, then the response of the function is invariant to the ordering of the elements of its input. This property is described as:

In our supervised video pedestrian retrieval task, it is given $t$ frame samples of $T^{1},\ldots,T^{t}$ as well as the person identity $y$. Since the frame features are fused using average pooling, shown in Fig. 1, thus it is obvious that the pedestrian identity predictor is permutation invariant to the order of frames in a clip (i.e., $f_{\theta}([T^{1},\ldots,T^{t}])=f_{\theta}([T^{\Pi(1)},\ldots,T^{\Pi(t)}])$ for any permutation $\Pi$). We continue to study the structure of the set function on countable sets and show that our set aggregation cell satisfies the structure of the set function.

In our deep architecture, we use an aggregation layer (i.e., set aggregation cell) to fuse frame features in a countable set (i.e., $|F|=t$), and this aggregation layer is a permutation invariant function. We use a simple case as an example, shown in Fig. 2(a). In this architecture, the $\gamma$ function is a mapping: $\mathbb{R}^{c\times t}\to\mathbb{R}^{c\times t}$, formulated as:

where ${\boldsymbol{G}}=[{\boldsymbol{g}}^{1},\ldots,{\boldsymbol{g}}^{t}]$ and ${\boldsymbol{F}}=[{\boldsymbol{f}}^{1},\ldots,{\boldsymbol{f}}^{t}]$. Thereafter, average pooling operates on the feature set, to realize the summation and $\beta$ function. Since the $\gamma$ and $\beta$ functions are all permutation invariant, the set aggregation cell is a valid set function. Similarly, in Fig. 2(b), the $\gamma$ function is realized as:

which also satisfies the condition of permutation invariance of its input. In the main paper, we evaluated the performance of two vanilla aggregation cells empirically and we observed that the aggregation cell with $\mathrm{Avg}$ function is superior to that with the $\mathrm{Max}$ function.

Since the $\mathrm{Avg}$ and the $\mathrm{Max}$ operations are permutation invariant, their summation is also permutation invariant; thus we continue to develop our set aggregation in the main paper, shown in Fig. 3. The $\gamma$ function is formulated as:

The above analysis shows the necessity to model the frame features in a clip as a set and that the set aggregation cell is a valid set function. In the main paper, we also verify the effectiveness of the set aggregation cell.

In the main paper, we evaluate our attention module on image person re-ID tasks on the CUHK01 [LiWei2012ACCVCUHK01] and the DukeMTMC-reID [ristani2016MTMC] datasets. The description of the two datasets is as follows:

We use a single query (SQ) setting for both datasets when calculating the network prediction accuracy.

In the main paper, we have evaluated our attention mechanism across four video person re-ID datasets and two image person re-ID datasets. Here, we show some samples from the aforementioned datasets, in Fig. 4 and Fig. 5. In each pedestrian bounding box, we use a black region to cover the face parts for the sake of privacy.

In this section, extensive experiments are performed to choose a proper setting for the baseline network, including the number frames to use from a video clip, dimensionality of the video feature embedding and the training strategies (e.g., pre-training and random erasing [zhong2017random]). This ablation studies are performed on the iLIDS-VID [WangTaiqing2016TPAMIiiLIDS-VID] and the MARS [ZhengLiang2014ECCVMAR] datasets.

In this section, we perform experiments for parameter selection in the set aggregation cell.

Fig. 6 shows additional visualizations of feature maps for qualitative study. In the visualizations, we can clearly observe that our attention module has the capacity to focus more on the foreground areas and ignore some background areas, which boosts the baseline network to achieve the state-of-the-art performance on the video pedestrian task.