MAFNet: A Multi-Attention Fusion Network
for RGB-T Crowd Counting

The development of crowd counting has gone through three stages: detection-based, regression-based, and density estimation-based methods.

Detection-based methods. The early crowd counting methods are mainly based on object detection [30, 31, 32, 33, 34, 35]. After the detection of individuals, the number of targets is counted to obtain the total number of people. Such algorithms perform well in scenes with sparse crowds and no occlusion, but poorly in dense crowds.

Regression-based methods. Due to the limitations of detection-based methods, researchers propose regression-based crowd counting methods [36, 37, 38]. These methods first extract global features or local features of the input image, and then use regression methods to learn a mapping function to obtain the crowd counts. However, the extracted low-level features may ignore the critical features of the targets and such methods cannot obtain the spatial distribution of the crowds, limiting the application in real scenes.

Density estimation-based methods. The methods based on density estimation [9, 10, 11, 39, 13, 40, 14, 41, 42, 15, 16, 18, 43, 44, 45] use spatial information to estimate the crowd density maps, which are the current mainstream crowd counting methods. The crowd density map not only reflects the spatial distribution of the crowds in the image, but also its integral value is the crowd counts. Zhang et al. [10] propose a Multi-column Convolutional Neural Network (MCNN) for crowd counting, which utilizes filters with different sizes of receptive fields to extract features. Li et al. [11] propose a Congested Scene Recognition Network (CSRNet) that uses dilated convolution in the backend to improve the receptive field and enhances feature extraction capability. Recently, some works improve the crowd counting performance by introducing multi-scale information [39, 13, 40] and perspective estimation [14, 41, 42]. To alleviate the problem of difficult collection and annotation of crowd counting datasets, some works [15, 16, 18] explore the domain-adaptive crowd counting from the synthetic datasets to the real-world. In addition, with the Vision Transformer (ViT) [3] first applying the transformer structure for vision tasks, many transformer-based [25] crowd counting methods [43, 44, 45] have been proposed with outstanding performance.

Multi-modal crowd counting aims at improving the performance of crowd counting methods by comprehending multi-modal data through multi-modal representation learning [46, 47, 48, 49, 50]. Multi-modal crowd counting methods mainly include three research directions: the methods based on RGB and depth images (RGB-D), based on auditory and visual information (Audio-Visual), and based on RGB and thermal images (RGB-T).

RGB-D crowd counting methods. The depth image reflects the scale information of the image, which facilitates the detection of small-scale targets for crowd counting. Bondi et al. [51] propose a RGB-D crowd counting dataset MICC and use depth images to segment people and localize candidate heads. Arteta et al. [52] introduce depth information and enhance crowd density estimation performance by foreground and background segmentation. Song et al. [53] use the Kinect sensor to obtain depth images and propose a depth region suggestion network for counting. Lian et al. [54], [55] propose a strategy to generate crowd depth-adaptive density maps, and a real-world RGB-D crowd counting dataset called ShanghaiTechRGBD and a synthetic dataset ShanghaiTechRGBD-syn are collected. Yang et al. [56] propose a Bidirectional Cross-modal Attention (BCA) mechanism to focus on crowded regions in images though depth information. Li et al. [57] fuse the RGB and depth images by cycle-attention and optimize the counting model from both fine pixel-aware and coarse pyramid region-aware perspectives. However, the depth information is easily disturbed and poorly robust to illumination changes and occlusions in practical applications.

Audio-Visual crowd counting methods. Researchs in the field of neurobiology have shown that visual and auditory information are widely used as perceptual mediators of human beings. Several researches introduce auditory information as an auxiliary cue for crowd counting tasks. Hu et al. [58] propose an estimation model that jointly learns visual and audio modalities, and release a large-scale audiovisual crowd counting dataset DISCO. Sajid et al. [59] propose an audio-visual multi-task network based on the transformer structure to achieve better pattern association and efficient feature extraction. Hu et al. [60] propose an Audio-Visual Multi-Scale Network (AVMSN) to model unconstrained visual and auditory sources for crowd counting. Nevertheless, auditory information is easily disturbed by the surrounding noise in a noisy scenario.

RGB-T crowd counting methods. Thermal images are insensitive to illumination variations and have strong ability to penetrate some particulate matters such as smog and fog, which could be used as complementary information to RGB images for crowd counting. Peng et al. [19] propose the DroneRGBT dataset, which is the first RGB-T crowd counting dataset based on drone view, and propose the MMCCN network to learn both modal information from RGB and thermal images simultaneously. Based on this, Zhang et al. [21] propose an improved model of MMCCN, the I-MMCCN network, which introduces a hard example mining module and a novel block average absolute error loss to the MMCCN network for performance enhancement. Besides, Liu et al. [20] propose a large-scale RGB-T crowd counting dataset RGBT-CC and a two-stream cross-modal representation learning framework. Tang et al. [22] propose a three-stream adaptive fusion network named TAFNet, which extracts the combination features of RGB and thermal images through an additional stream. After introducing the thermal information, the performance of crowd counting methods in low-illumination and similar backgrounds is effectively improved. However, the existing RGB-T crowd counting methods use CNN-based cross-modal fusion structure, which makes it difficult to obtain long-range contextual information across modalities due to the small perception field. Unlike the previous methods, we adopt a multi-modal fusion method based on the attention mechanism, which can effectively focus on the global and contextual information between the modalities.

In the encoder part, the VGG19 [29] networks which discard the fully-connected layers are used as the backbones to extract multi-scale features of the RGB and thermal modalities. The backbones are divided into 5 stages and the last layer of each stage is the max-pooling layer with a stride of 2 for down-sampling. To fully interact information of the two modalities, an intermediate fusion strategy is adopted by embedding the proposed MAF modules into the adjacent stages of the backbones. Considering that if the information interaction of large-scale features between the two modalities is carried out at the shallow stages, the fusion effect may be affected by the misalignment of the paired RGB-T data. In our implementation, the MAF modules are embedded between deep stages to fuse the features with small-scale but at the high semantic level, which alleviate the data misalignment problem through the translation invariance of the max-pooling layer [20] in the feature extraction stage. More specifically, the RGB and thermal images are fed to the two backbones respectively for modality-specific representation learning in stage 1 and 2. Starting from stage 3, the features are first further extracted through the backbones, and then fed to the MAF modules for information interaction and enhancement. Finally, three paired feature maps from the two modalities are obtained from the output of MAF blocks. Their scales are $1/8$, $1/16$, $1/32$ of the input images, and the corresponding number of channels are 256, 512, 512, respectively.

On account of the small perception field of convolution operations, the existing CNN-based multi-modal fusion modules have difficulty in adequately capturing long-range contextual information from the RGB and thermal modalities. Considering the powerful cross-modal global modeling capability of the attention mechanism, an attention-based Multi-Attention Fusion (MAF) module is proposed for better information interaction between the two modalities. The implementation of the MAF module represents in Fig. 3-(a).

Patch Embedding. To use the attention mechanism, the input 2D feature maps should first be converted into a sequence consisting of patches. Given a feature map $F_{m}\in\mathbb{R}^{C\times H\times W}$ of modal $m$ with the size of channel $C$, height $H$ and width $W$ and reshaped it into a sequence composing of $N$ patches of size $P$ with the dimension of $P^{2}\times C$, where $(N=HW/P^{2})$. Then the patches are flattened and mapped to D-dimensional latent vectors via a trainable linear projection layer. The outputs of the projection layer are called patch embeddings, which can be formulated as:

where $x_{p}^{i}$ means the $i$-th flattened patch ($i=1,2,...,N$) and $E$ $(E\in\mathbb{R}^{(P^{2}\times C)\times D})$ denotes the linear projection matrix of the embedding process. In our implementation, the patch size of the first MAF module is set to 2 and the last two are set to 1. The embedding dimension $D$ is set to 768.

MAF block. To fully capture the long-range information within each modality and across modalities by the attention mechanism, the MAF block consisting of Intra-Modality-Attention (IMA) module and Cross-Modality-Attention (CMA) module is proposed. By stacking a suitable number of MAF blocks, the MAF module have a powerful cross-modal fusion capability at the global level. In this part, we first review the Multi-Head Attention mechanism and then introduce the implementation of the IMA and CMA modules.

(a) Multi-Head Attention mechanism. First, we introduce the implementation of Multi-Head Attention mechanism [25]. Given two patch embeddings $z_{1}$ and $z_{2}$ as inputs. Multi-Head Attention executes multiple independent attention heads in parallel and concatenates the outputs, which is then projected to obtain the final value. The input of one attention head $h$ is a triplet $Q_{1}^{h}$(query), $K_{2}^{h}$(key) and $V_{2}^{h}$(value) calculated from the input embeddings:

where $W^{h}_{q}$, $W^{h}_{k}$, $W^{h}_{v}\in\mathbb{R}^{D\times d}$ are the three learnable linear projection matrices and $d$ is the dimension of the channels. And one attention head is formulated as:

where $Y^{h}$ terms the output of one attetnion head. Then the Multi-Head Attention is expressed by the following formula:

where $z$ is the output of the Multi-Head Attention processing. $N_{h}$ is the number of attention heads, $Concat[...]_{h=1}^{N_{h}}$ means the concatenation operation for all attention heads and $W^{O}\in\mathbb{R}^{hd\times N}$ is the linear projection matrices. In summary, the processing of the Multi-Head Attention is presented as follows:

(b) Intra-Modality-Attention (IMA) module and Cross-Modality-Attention (CMA) module. Based on the multi-head attention mechanism, the IMA and the CMA module are designed as shown in Fig. 3-(b, c), respectively. Specifically, in the $l$-th MAF block $(l=1,2,..,l_{N})$, given the input patch embeddings $z_{r}^{l-1}$ and $z_{t}^{l-1}$ of different modalities. The input query, key and value of the IMA module are from the same modality for the intra-modal features enhancement. Formally:

where $\{...\}^{i}$ indicates the variable using in IMA module. $IMA_{r\to r}$ means the operation is performed within the RGB modality and $IMA_{t\to t}$ means the operation is performed within the thermal modality. Different from the IMA module, the input key and value of the CMA module are from the same modality but the query is from the other. The CMA module pays attention to the regions of interest across modalities for feature alignment and cross-modal information interaction by means of attention maps computed by the query and key of different modalities. Formally:

where $\{...\}^{c}$ indicates the variable using in CMA module. $CMA_{t\to r}$ means that the query is obtained from thermal modality, and key and value are obtained from RGB modality for co-attention. In contrast, $CMA_{r\to t}$ means that the query is from RGB modality, and key and value are from thermal modality. Then, the features obtained from different sub-modules are aggregated by element-wise product in different modalities. The aggregated process can be formulated as:

Finally, the features obtained from the last MAF block of a MAF module termed $z_{r}^{l_{N}}$ and $z_{t}^{l_{N}}$, where $l_{N}$ means the number of MAF blocks in one MAF module. In our implementation, the $l_{N}$ in each MAF module is taken as 2, i.e. stacking two MAF blocks.

Recovering 2D structure and skip connection. In order to obtain the 2D feature maps for the next stage of feature extraction, the final output features of the last MAF module should be mapped back to the dimension of the input embeddings by a linear projection layer and recovered the 2D structure. Besides, in consequence of the internal complexity of the MAF module may degrade the performance of the network, a skip connection structure is introduced to obtain the final output feature map of this stage by directly summing the input and output feature maps inspired by ResNet [61].

Considering that the output feature maps from the encoder may lose crucial information in the process of down-sampling and the feature maps with different scales may focus on different levels of semantic information from different modalities, a Multi-modal Multi-Scale Aggregation (MMA) regression head is proposed to aggregate the multi-modal and multi-scale feature information to reconstruct detail information and generate high-quality crowd density maps. Specifically, the paired feature maps of the two modalities with the same scale output from the the MAF modules are first concatenated by channel and then fed to the convolutions of $3\times 3$ to fully aggregate information of RGB and thermal modalities. Then the multi-scale features are up-sampled to the same scale and added together. Following, the added features are fed into four parallel branches, three of which contain dilated convolutions of $3\times 3$ with dilation rates of 1,2,3 for increasing the perceptive fields and one skip connection with $1\times 1$ convolution. The output features of the top three branches are concatenated and added with the skip connection together to obtain the final aggregated feature maps. Finally, the aggregated feature maps are fed to a $3\times 3$ convolution layer and a $1\times 1$ convolution layer to generate the predicted crowd density map.

For training the proposed model, we adopt the standard Mean Squared Error (MSE) loss function, which uses Euclidean distance to calculate the gap between the ground truth and the predicted crowd density map. The loss function can be formulated as:

where $N$ is the number of training samples, $i$ means the $i$-th training sample, $\hat{D}_{i}$ represents the predicted density map and $D_{i}$ represents the density map generated by ground truth.

RGBT-CC [20] is a large-scale RGB-T crowd counting dataset for surveillance view, which contains 2030 pairs of RGB-T images with the size of 640×480 consisting of 138,389 pedestrians. Among the 2030 images, 1013 images are captured under light conditions and 1017 images are captured under dark conditions.

DroneRGBT [19] is a RGB-T crowd counting dataset for drone view, which contains 3600 pairs of RGB-T images with the size of 640×512 consisting of 175,698 pedestrians. Among them, about 1600 images were captured under dusk conditions, 1300 images under light conditions, and 900 images under dark conditions.

Implementation details. The experiments are implemented on NVIDIA GeForce RTX 3090 GPU with 24GB memory. In the training phase, we adopt an optimizer of AdamW, the initial learning rate of $5\times 10^{-5}$ with a linear warmup strategy, a batch size of 16 and a maximum number of iterations of 300. Normal data augmentation is applied to the input images, including rescale, random crop and flip. The input images are randomly cropped to the size of 256$\times$256. To convert the point annotations into the corresponding crowd density maps, we use Gaussian convolution with kernel size of 7 and variance $\sigma$ of 2 to extend the sparse points. In addition, our code is built on $C^{3}$-Framework [62].

Evaluation Metrics. We use Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) as evaluation metrics in the two datasets. MAE and RMSE are defined as:

where $\hat{P}_{i}$ and $P_{i}$ refer to the predicted count and ground truth of the $i$-th RGB-T image pairs, respectively. In addition, following [20], Grid Average Mean Absolute Error (GAME) is utilized in the RGBT-CC dataset to evaluate the performance of different regions. Specifically, for a particular level $l$, the image is grided into $4^{l}$ non-overlapping regions to calculate the counting error of each region and sum up. GAME is defined as:

where $\hat{P}_{i}^{j}$ and $P_{i}^{j}$ refer to predicted count and ground truth of the $j$-th part of the $i$-th RGB-T image pairs, respectively. It is worth noting that GAME(0) corresponds to the counting error of the whole image, which is equivalent to the MAE.

In order to verify the effectiveness of each component in the proposed Multi-Attention Fusion Network (MAFNet), extensive ablation studies are performed on the RGBT-CC dataset. A model with a dual VGG19 encoder and a regression head consisting of a $3\times 3$ convolution and a $1\times 1$ convolution is used as the baseline. We gradually add the proposed modules to the baseline to obtain variant models and compare the performance to the baseline to verify the effectiveness of each module. The evaluation results are shown in Table I, and row 1 shows the result of the baseline.

Effectiveness of Multi-Attention Fusion module. The Multi-Attention Fusion (MAF) module is stacked by MAF blocks, which consists of two Intra-Modality-Attention (IMA) modules and a Cross-Modality-Attention (CMA) module. To demonstrate the effectiveness of the MAF module, we perform ablation studies both on the IMA and CMA modules. For results of the row 2 and row 3, the models with the IMA and CMA modules, respectively, both perform better than the baseline in all evaluation metrics, which verify the effectiveness of the IMA and the CMA module. Especially, the performance improvement of the model with the CMA module is much higher than that of the model with the IMA module. That’s because the cross-modal information interaction relies mainly on the CMA module, which plays a more important role in the MAF module. Compared with the results in row 5 and row 6, the model with CMA also performs better than the model with IMA, further illustrating the importance of CMA in cross-modal information interaction. As shown in row 7, the model with both IMA and CMA modules exhibits satisfactory performance, which is close to the performance of the full structure of MAFNet (row 7 v.s. row 8).

Effectiveness of Multi-modal Multi-scale Aggregation regression head. The Multi-modal Multi-Scale Aggregation (MMA) regression head is proposed to aggregate the multi-modal and multi-scale feature information to predict crowd density maps. Compared with row 4 and the baseline shown in row 1, the performance of the model is improved after adding the MMA module to the baseline model, which verifies the effectiveness of the MMA regression head. Further, the performance improvement of the MMA module combined with the IMA and CMA modules respectively are explored as shown in row 5 and row 6. The performance of the models with IMA and CMA modules respectively are both greatly improved on all evaluation metrics with the addition of MMA regression head, where GAME (0) are improved 5.10$\%$ / 6.56$\%$ (row 2 v.s. row 5 and row 3 v.s. row 6). From this, it could be observed that the model using the IMA module has a better improvement than the model with the CMA module after adding the MMA regression head. Considering that IMA module cannot implement cross-modal information interaction in the encoder stage, the multi-modal information is aggregated in the regression stage by introducing MMA to compensate for the limitation of IMA. Through the above extensive ablation studies, the effectiveness of the MMA regression head in aggregating multi-modal and multi-scale information in MAFNet is well demonstrated.

Table II(b) lists the performance comparison between the MAFNet and four state-of-the-art methods on two datasets respectively and it could be found that the MAFNet shows strong performance, and outperforms the existing methods. As shown in Table II(b)-(a), on the RGBT-CC dataset, MAFNet improved 11.95$\%$, 14.96$\%$, 16.01$\%$, 20.47$\%$, and 7.26$\%$ in the five metrics of GAME(0), GAME(1), GAME(2), GAME(3) and RMSE respectively, compared with the state-of-the-art method TAFNet [22]. And as shown in Table II(b)-(b), compared with I-MMCCN [21], MAFNet achieves 7.38$\%$ and 9.77$\%$ improvement on MAE and RMSE. In addition, existing methods such as CMCRL [20] and TAFNet [22] do not utilize multi-scale information, and the performance of these models are completely dependent on the proposed multi-modal fusion modules. In fairness, we use the MAFNet without MMA regression head to compare with the state-of-the-art methods (row 4 v.s. row 5 in Table II(b)-(a) and (b)) and the experiments show that the MAFNet without MMA regression head still outperforms the state-of-the-art methods, which demonstrates the superiority of the designed MAF module over the existing multi-modal fusion modules. A visualization comparison of the partial state-of-the-art methods with MAFNet regarding the generated density maps for different density levels on the two datasets are shown in Fig. 4. The visualization results show that MAFNet could predict more accurate counting results and perform better in dense crowd scenes than the state-of-the-art methods.

To further illustrate the excellent robustness of MAFNet for illumination conditions, the RGBT-CC test set of 800 images is divided into 406 brightness and 394 darkness images to test the model’s performance in different illumination conditions. As the results shown in Table III, comparing with CMCRL [20] and TAFNet [22], the MAFNet outperforms these methods under different illumination conditions. It is worth noting that the MAFNet exhibits similar performance under different illumination conditions, while the other methods are much different. This phenomenon precisely shows that MAFNet is more robust to illumination variations than other methods.

Visualization analysis. To further illustrate the effectiveness of the proposed model, the attention maps of the three MAF modules are visualized in Fig. 5. From this, it is clear that the network’s attention is indeed focused on the target crowds, which illustrates the effectiveness of the attention mechanism used in the MAF module. Further, by comparing the attention maps obtained in different MAF modules, it could be observed that the MAF modules embedded at different locations focus on different levels of feature information. The shallow MAF module is better for head focus in sparse scenes because the inputs are large-scale feature maps, while it is difficult to fully focus on the entire crowd in more dense scenes. In contrast, the deep MAF module uses semantics-rich and small-scale feature maps that could model global information well for dense scene detection, but the performance of specific target recognition in sparse scenes decreases due to the low resolution. In particular, the second MAF module embedded in the middle stage has well-focused attention to crowds in different scales and density scenes. By observing the visualized results, it is a natural idea to introduce the MMA regression head to make full use of the multi-modal and multi-scale information for improving the robustness of the model to different density scenes.

Effectiveness of RGB-T multi-modal data. To verify the effectiveness of multi-modal data, RGB and thermal images are used to train the proposed model respectively and compared the performance with the RGB-T trained model. As shown in Table 6, using only RGB data to train the model works unsatisfactorily (row 1) because it is difficult to discriminate the people in RGB images under dark conditions, which provides insufficient information for crowd counting tasks. In comparison, the performance of the model trained with thermal images (row 2) is much better and closer to the performance of the model trained with RGB-T data (row 9), indicating that thermal images play a more important role in RGB-T crowd counting tasks, especially in dark environments. To further illustrate the necessity of multi-modal data, some extreme examples are listed in Fig. 6. In the first row, Fig. 6-(a) shows the RGB image with low-illumination condition, which leads to large error in the predicted crowd density map shown in Fig. 6-(d). In the last row, the thermal image shown in Fig. 6-(b) is poorly imaged with the unclear outline of the people in the boxed region leading to huge counting errors shown in Fig. 6-(e). Thanks to the high complementarity of RGB and thermal images, both of the modalities compensate for the information of the other to ensure the quality of the predicted crowd density maps, which are shown in Fig. 6-(f).

Patch Size. In the conventional vision transformer models, using large size patches such as 16×16 in ViT [3] will only result in coarse-grained features, which may cause large errors for intensive prediction tasks such as crowd counting. In contrast, using small size patches will make the sequence so long that it will cause a large amount of computation. However, as mentioned in Section III-A, the MAF modules are only embedded in the high stages to process the small-scale features without causing much computation. Therefore, the patch size of the first MAF module is set to 2 and the last two are set to 1 to ensure both the acquisition of fine-grained features and low computation in our implementation.

The necessity of Position Embedding. Although position embedding is the default component of the attention-based transformer model [25], it is worth discussing whether it is necessary in the MAF module. The comparison results of whether to add position embedding are shown in Table I and $PE$ means the position embedding. It could be found that the two methods get comparable performance. Specifically, the network with position embedding performs poorly in GAME(0) (10.95 vs 10.90) but performs better in RMSE (19.85 vs 20.82). Considering that both convolution and attention mechanisms are used in the network, the skip connection of the MAF module somewhat introduces the inductive bias of convolution, which provides the location information for the attention calculation. Therefore, the addition of position embedding has little effect on the model’s performance.

Comparison of different numbers of MAF modules. The number of MAF modules is an important hyperparameter. Assuming that the MAF modules are also embedded in the earlier stages, on the one hand the computation will increase, and on the other hand the additional counting errors will be introduced due to the paired image unalignment, which has mentioned in Section III-A. To find the optimal number of MAF modules, the MAF modules are embedded after each stage starting from the last stage of the backbone in turn. As shown in Table VI, the performance of the network is gradually getting better by increasing the number of MAF modules and the best result is obtained when the number of MAF modules is 3. When increasing to 4 modules, the model’s performance starts to degrade. Eventually, the number of the MAF modules is set to 3 in the network.

Comparison of different depths of MAF module. The depth of MAF module means the number of stacked MAF blocks in MAF module, which is also an important hyperparameter. We test different configurations of the network that contains different depth of MAF modules and show the results in Table VI. The 3 numbers in the second column of Table VI correspond to the depth of each of the 3 MAF modules, which indicates the depth of the MAF module greatly affects the performance of the model. Considering that the performance of MAE(GAME(0)) and RMSE is more important than other evaluation metrics, we set the depth of all three MAF modules to 2 as our final implementation to get the best result.