RGB-T Multi-Modal Crowd Counting
Based on Transformer

Crowd counting can be achieved by detection [Hoai and Zisserman(2014), Stewart et al.(2016)Stewart, Andriluka, and Ng, Idrees et al.(2015)Idrees, Soomro, and Shah, Lian et al.(2021)Lian, Chen, Li, Luo, and Gao] or density map estimation [Wang and Breckon(2022), Jiang et al.(2020)Jiang, Zhang, Zhang, Lv, Zhou, Pang, Xu, and Xu, Wang et al.(2022c)Wang, Cai, Han, Zhou, and Gong, Chen et al.(2022)Chen, Wang, Su, and Wang]. Since the latter can solve high overlap and occlusion problem, it shows better performance than the former.

The large-scale variation generated by the wide viewing angle of cameras and 2D perspective projection is a major challenge in crowd counting. The persons which are close to the camera are large, while the persons which are far from the camera are small. Multi-scale architecture[Zhang et al.(2016)Zhang, Zhou, Chen, Gao, and Ma, Babu Sam et al.(2017)Babu Sam, Surya, and Venkatesh Babu, Li et al.(2018)Li, Zhang, and Chen, Yuan et al.(2020)Yuan, Qiu, Liu, Wu, Chen, Chen, and Lin, Bai et al.(2020)Bai, He, Qiao, Hu, Wu, and Yan, Dai et al.(2021)Dai, Liu, Ma, Zhang, and Zhao] and perspective information[Shi et al.(2019)Shi, Yang, Xu, and Chen, Yan et al.(2019)Yan, Yuan, Zuo, Tan, Wang, Wen, and Ding, Yang et al.(2020b)Yang, Li, Wu, Su, Huang, and Sebe, Gao et al.(2019)Gao, Wang, and Li, Yang et al.(2020a)Yang, Li, Du, Huang, and Sebe] are two main solutions. Recently, to deal with the scale changes, some attention based methods are proposed. MAN[Lin et al.(2022)Lin, Ma, Ji, Wang, and Hong] improves global attention in the transformer by adding region attention. HANet[Wang et al.(2022b)Wang, Sang, Wu, Liu, and Sang] introduces scale context in the parallel spatial attention and channel attention.

In the paper, we solve the large-scale variation problem by multi-scale transformer based on tokens. The original token sequence is merged into a middle-scale token sequence and a large-scale token sequence, respectively. Then the three are parallel handled by three multi-head self-attention structures. Finally, three branches are concatenated and combined. The multi-scale concept ensures abundant receptive fields which benefits the crowd counting task.

Previous works utilize the convolution neural network as the backbone and regress density map to predict the crowd count. The advent of transformer has pushed the crowd counting model forward. BCCTrans [Sun et al.(2021)Sun, Liu, Probst, Paudel, Popovic, and Van Gool] introduces a global context learnable token to guide the counting. SAANet [Wei et al.(2021)Wei, Kang, Yang, Qiu, Shi, Tan, and Gong] designs a deformer backbone to extract the features, aggregates multi-level features by a deformable transformer encoder, and introduces a count query in a transformer decoder to re-calibrates the multi-level feature maps. DCSwinTrans[Gao et al.(2022)Gao, Gong, and Li] enhances the large-range contextual information by a dilated Swin Transformer backbone, and equips with a feature pyramid networks decoder to achieve crowd instant localization. CrowdFormer [Yang et al.(2022)Yang, Guo, and Ren] models the human top-down visual perception mechanism by an overlap patching transformer block. CCTrans [Tian et al.(2021)Tian, Chu, and Wang] adopts a pyramid transformer and a multi-scale regression head to achieve both fully-supervised and weakly-supervised crowd counting task. In addition, in weakly-supervised crowd counting, there are some other transformer based methods. TransCrowd [Liang et al.(2022)Liang, Chen, Xu, Zhou, and Bai] uses a learnable counting token or global average pooling on high-layer semantic tokens to represent the crowd numbers. It constructs a weakly supervised model from sequence-to-count perspective. SFSL [Chen and Lu(2022)] introduces a learnable unbiased feature estimation of persons and utilizes the feature similarity for the regression of crowd numbers to solve the lack of local supervision. CrowdMLP [Wang et al.(2022d)Wang, Zhou, Cai, and Gong] proposes a multi-granularity multilayer perceptron (MLP) regressor to enlarge receptive fields and a split-counting to decouple spatial constraints. JCTNet [Wang et al.(2022a)Wang, Liu, Long, Sang, Xia, and Sang] introduces transformer structure upon the high-layer feature of convolutional neural network and regresses the count.

In the paper, we use transformer encoder structure to achieve count-guided multi-modal fusion, and use transformer decoder structure to perform modal-guided count enhancement.

Although the crowd counting methods have achieved many significant improvements, they rely on optical information and often perform poorly when the light is insufficient. To solve this problem, RGB-T crowd counting has been getting a lot of attentions. On one hand, thermal image can recognize pedestrians in poor illumination conditions. On the other hand, thermal image can reduce wrong recognition about some human-shaped objects. Meanwhile, RGB image can suppress interference in thermal images. For example, heating walls and lamps that are highlighted in thermal images can be filtered from color perspective. Therefore, RGB and thermal images need to be simultaneously explored.

CMCRL [Liu et al.(2021)Liu, Chen, Wu, Li, Li, and Lin] introduces a two-stream framework that first aggregates two features and second propagates the common information to further refine each feature. TAFNet [Tang et al.(2022)Tang, Wang, and Chau] uses a three-stream network to learn the RGB feature, the thermal feature, and the concatenated RGB-T feature for crowd counting. The proposed Information Improvement Module (IIM) is used to fuse the modal-specific and combination features. Mutual Attention Transformer (MAT) [Wu et al.(2022)Wu, Liu, Zhang, Mao, Lin, and Li] uses cross-modal mutual attention to build long-range dependencies and enhance semantic features in crowd counting task. DEFNet [Zhou et al.(2022)Zhou, Pan, Lei, Ye, and Yu] uses multi-modal fusion, receptive field enhancement, and multi-layer fusion to highlight the crowd position and suppress the background noise.

In these works, the fusion of the RGB and thermal images are short of count objective constraint. We design a learnable count token to guide multi-modal fusion.

Given a paired RGB-T image $I=\{I_{r},I_{t}\}$, we use two PVT encoders [Wang et al.(2021)Wang, Xie, Li, Fan, Song, Liang, Lu, Luo, and Shao] as the feature extractors to capture hierarchical features.

	$\displaystyle F_{r}=\mathcal{E}_{PVT}(I_{r})$		(1)
	$\displaystyle F_{t}=\mathcal{E}_{PVT}(I_{t})$

where $\mathcal{E}_{PVT}$ denotes a PVT encoder, $F_{r}=\{F_{r}^{i}\}_{i=1}^{4}$ and $F_{t}=\{F_{t}^{i}\}_{i=1}^{4}$ represent color features and thermal features, respectively, $i$ is the feature layer number.

The high-layer features contain more semantic information, which are suitable to obtain the global counting cues. Besides, color feature and thermal feature have each advantage in representing the crowd. Therefore, we use the high-level tokens from color modality and thermal modality to excavate the number of crowd. To fully align two-modal data and generate a consistent result, a learnable count token is designed to guide the two-modal fusion. Specifically, as is illustrated in Fig.2, high-layer semantic features $F_{r}^{4}$ and $F_{t}^{4}$ are generated from color encoder and thermal encoder, respectively. They represent unaligned multi-modal semantic concept. We design a learnable count token $F_{count}$ which implies the coarse number of crowd. The three are concatenated along the token direction, and then fed into a Multi-Scale Token Transformer (MSTTrans) which spreads information among color, thermal, and crowd count by the multi-head self-attention.

MSTTrans is proposed to solve large-scale variations. Inspired by multi-scale design in atrous spatial pyramid pooling (ASPP) [Chen et al.(2017)Chen, Papandreou, Kokkinos, Murphy, and Yuille], MSTTrans achieves multi-scale transformer based on tokens. At first, we concatenate high-layer color feature, high-layer thermal feature, and the learnable count token to form an initial token sequence. Then, we merge the initial token sequence to generate a middle-scale token sequence. The middle-scale token sequence has the larger receptive fields than original token sequence. Besides, we merge the initial token sequence to generate a large-scale token sequence, where a modality is represented by a token. According to above merge strategy, three parallel branches which all include color modality, thermal modality, and count token are constructed. They are fed into three multi-head self-attention modules for in-depth fusion.

Specifically, as is illustrated in the right of Fig. 2, suppose the high-layer semantic feature $F_{r}^{4}\in\mathbb{R}^{N^{2}\times C}$ and $F_{t}^{4}\in\mathbb{R}^{N^{2}\times C}$, where $N^{2}$ and $C$ represent the number of tokens and channels, respectively. The two-modal features and the learnable count token are concatenated to generate the initial token sequence $f_{1}\in\mathbb{R}^{(2N^{2}+1)\times C}$.

$\displaystyle f_{1}=[F_{r}^{4},F_{t}^{4},F_{count}]$

(2)

where $[\cdot]$ denotes concatenation operation along token direction.

Then, the two-modal features are merged to $N$ groups and each group generates $N$ middle-scale tokens. All the middle-scale tokens and the learnable count token are concatenated to generate the middle-scale token sequence $f_{2}\in\mathbb{R}^{(2N+1)\times C}$.

$\displaystyle f_{2}=[\textit{merge}_{N^{2}\rightarrow N}(F_{r}^{4}),\textit{merge}_{N^{2}\rightarrow N}(F_{t}^{4}),F_{count}]$

(3)

where $\textit{merge}_{a\rightarrow b}$ denotes the aggregation operation from $a$ tokens to $b$ tokens which applies a reshape operation and a fully connected layer.

Meanwhile, the two-modal features are merged to two groups and each group generates a large-scale token. The large-scale tokens and the learnable count token are concatenated to generate the large-scale token sequence $f_{3}\in\mathbb{R}^{(2+1)\times C}$. There are a color token, a thermal token, and a learnable count token. It ensures two-modal whole alignment under the guidance of count.

$\displaystyle f_{3}=[\textit{merge}_{N^{2}\rightarrow 1}(F_{r}^{4}),\textit{merge}_{N^{2}\rightarrow 1}(F_{t}^{4}),F_{count}]$

(4)

Three token sequences with different scales are fed into three multi-head self-attention modules for multi-modal interaction.

$\displaystyle f_{i}^{\prime}=\textit{MHSA}(f_{i})$

(5)

where MHSA represents two multi-head self-attention layers.

Since the lengths of middle-scale and large-scale token sequences are different from initial token sequences, we apply fully connection layer and reshape operation to restore token sequence length.

$\displaystyle g_{i}=\textit{Reshape}(\textit{FC}(f_{i}^{\prime}))$

(6)

where $i=2,3$ because only middle-scale and large-scale token sequences should be restored, FC is a fully-connected layer, and Reshape is a reshape operation to restore token length.

Further, to retain the original features in the middle-scale and large-scale branches, the concatenation and MLP operations are successively conducted.

$\displaystyle g_{i}^{\prime}=\textit{MLP}(\textit{Concat}(g_{i},f_{1}))$

(7)

where $i=2,3$, Concat is concatenation operation along channel direction, and MLP is a two-layer perceptron.

Last, three features are concatenated and shrunk in channels.

$\displaystyle G=[G_{r},G_{t},G_{count}]=\textit{MLP}(\textit{Concat}(f_{1}^{\prime},g_{2}^{\prime},g_{3}^{\prime}))$

(8)

where $G$ has the same size as the input $f_{1}$ of MSTTrans module, and consists of optimized color feature $G_{r}$, thermal feature $G_{t}$, and count feature $G_{count}$.

In MSTTrans module, the count token is responsible for incorporating the global information and perceiving the number of persons. Besides, it is used to guide the fusion of color feature and thermal feature. Under the guidance of count token, color feature and thermal feature are in-depth interacted. Moreover, multi-scale token concept ensures the abundant receptive fields adaptive to recognizing the persons with different sizes.

The researches pointed out that the thermal image can provide strong support on density map estimation, especially in the dark background[Tang et al.(2022)Tang, Wang, and Chau]. In the paper, we use the thermal modality to predict the density map and count, and further use color modality to refine the prediction.

Therefore, after the previous count-guided multi-modal fusion, we design a modal-guided counting enhancement module which is responsible for generating the density map and final count from one modality under the guidance of the other modality. A multi-scale deformable transformer (MSDTrans) is employed to achieve the above objective.

Specifically, the thermal feature $G_{t}$ and the learnable count token $G_{count}$ are concatenated as query ($Q$), and the enhanced color feature $G_{r}$ and the encoded low-layer features $F_{r}^{i}(i=1,2,3)$ compose multi-scale color features which are regarded as key ($K$) and value ($V$). We use multi-scale deformable attention [Zhu et al.(2020)Zhu, Su, Lu, Li, Wang, and Dai] to enhance $Q$ by $K$ and $V$. Last, it will output modal-guided enhanced feature $O_{t}$ and count token $O_{count}$.

$\displaystyle\left[O_{t},O_{count}\right]=\textit{DeformAttn}(\left[G_{t},G_{count}\right],\{G_{r},F_{r}^{3},F_{r}^{2},F_{r}^{1}\})$

(9)

where $\textit{DeformAttn}(a,b)$ is the multi-scale deformable attention [Zhu et al.(2020)Zhu, Su, Lu, Li, Wang, and Dai], $a$ represents content feature, $b$ is multi-scale features.

To obtain the density map, we use a simple regression head which consists of two 3$\times$3 convolution layers and one 1$\times$1 convolution layer.

$\displaystyle D=\textit{RH}(O_{t})$

(10)

where RH is the regression head.

The loss includes a loss about the density map and a loss about the learnable count token.

$\displaystyle\mathcal{L}=\mathcal{L}_{D}(D,D^{\star})+\mathcal{L}_{C}(O_{count},C^{\star})$

(11)

where $\mathcal{L}_{D}$ adopts distribution matching loss proposed in[Wang et al.(2020)Wang, Liu, Samaras, and Nguyen], which supervises the density map regression and count estimation, $\mathcal{L}_{C}$ adopts $L_{1}$ norm ($\|\cdot\|_{1}$) to supervise the count token. $D^{\star}$ and $C^{\star}$ represent the ground truth of density map and count, respectively.

Dataset. The public RGBT-CC[Liu et al.(2021)Liu, Chen, Wu, Li, Li, and Lin] dataset is adopted to evaluate our method. RGBT-CC consists of 1,030 training samples, 200 validation samples, and 800 testing ones.

Evaluation Metrics. The widely used Grid Average Mean Absolute Error (GAME)[Guerrero-Gómez-Olmedo et al.(2015)Guerrero-Gómez-Olmedo, Torre-Jiménez, López-Sastre, Maldonado-Bascón, and Onoro-Rubio] and Root Mean Square Error (RMSE) are used as evaluation metrics[Liu et al.(2021)Liu, Chen, Wu, Li, Li, and Lin, Tang et al.(2022)Tang, Wang, and Chau].

$\displaystyle GAME(l)=\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{4^{l}}\mid\hat{P}^{j}_{i}-P^{j}_{i}\mid$

(12)

where $\hat{P}^{j}_{i}$ represents the predicted value of the $j^{th}$ region of the $i^{th}$ image, $P^{j}_{i}$ indicates the ground truth corresponding to $\hat{P}^{j}_{i}$, $4^{l}$ means the number of the divided non-overlapping regions of the image, and $N$ is the number of paired images in testing dataset. GAME sums the counting errors in all the regions.

$\displaystyle RMSE=\sqrt{\frac{1}{N}\sum_{i=1}^{N}(\hat{P}_{i}-P_{i})^{2}}$

(13)

where $\hat{P}_{i}$ represents the predicted value of the $i^{th}$ image, $P_{i}$ indicates the ground truth corresponding to $\hat{P}_{i}$. For both RMSE and GAME, lower value means the better performance.

The implementation setting includes: (1) GPU (NVIDIA RTX 3090); (2) input image size ($224\times 224$); (3) train time (17 hours); (4) learning rate ($1e-5$); (5) weight decay ($1e-4$).

To make quantitative comparisons, our method is compared with recent prominent approaches, including CSRNet[Li et al.(2018)Li, Zhang, and Chen], BL[Ma et al.(2019)Ma, Wei, Hong, and Gong], DM-Count[Wang et al.(2020)Wang, Liu, Samaras, and Nguyen], P2PNet[Song et al.(2021)Song, Wang, Jiang, Wang, Tai, Wang, Li, Huang, and Wu], MARUNet[Rong and Li(2021)], MAN[Lin et al.(2022)Lin, Ma, Ji, Wang, and Hong], CMCRL[Liu et al.(2021)Liu, Chen, Wu, Li, Li, and Lin], TAFNet [Tang et al.(2022)Tang, Wang, and Chau], MAT[Wu et al.(2022)Wu, Liu, Zhang, Mao, Lin, and Li], and DEFNet[Zhou et al.(2022)Zhou, Pan, Lei, Ye, and Yu] which are shown in Table 1. The top of the table shows six single-modal crowd counting models which are retrained by the input fusion of RGB and thermal images. The bottom of the table shows four RGB-T crowd counting models and ours. From the observation, we can conclude our method performs the best among all the methods. It achieves about $8.4\%$, $7.8\%$, $5.7\%$, $4.1\%$, $10.9\%$ improvement over the second best result in GAME(0), GAME(1), GAME(2), GAME(3) and RMSE, respectively. The great improvement profits from the multi-modal fusion under the guidance of count token and count enhancement of a modality under the guidance of the other modality.