3D Random Occlusion and Multi-Layer Projection for Deep Multi-Camera Pedestrian Localization

A recent survey on multi-view pedestrian detection can be found in [18]. The state-of-the-art methods in this field can be categorised into top-down approach and bottom-up approach. The top-down approach divides the ground plane into a grid. Each location in this grid is thought of as the location of a potential pedestrian and is back-projected to individual views for finding the optimal match between foregrounds and a generative model. The bottom-up approach projects the foregrounds from the individual views to a reference view and analyses the overlaid foreground projections to determine the locations of pedestrians.

Top-Down Approach Fleuret et al. [9] estimated a probabilistic occupancy map through a generative model that represents each pedestrian as a filled rectangle of the average size of pedestrians. The occupancy probability was updated iteratively for finding the locations of the rectangles which cover more foreground pixels in all the views. On the basis of this point of view, Alahi et al. [4] formulated multi-view pedestrian detection as a linear inverse problem; Peng et al. [17] modelled pedestrians and their occlusion relationships by using a multi-view Bayesian network; Yan et al. [25] used a non-iterative logic minimization method to reduce false-positive detections. Chavdarova and Fleuret [7] proposed an end-to-end multi-view pedestrian detection network. They back-projected each ground-plane location to individual views and created a rectangle box at the corresponding positions. A CNN was used to extract features within these rectangles and infer the locations of pedestrians by using Multi-Layer Perception. Baqué et al. [5] proposed a method which combines CNNs and a Conditional Random Field. The CNN in the discriminative model extracts pedestrian features from individual views and uses Gaussian Mixture networks to classify the body parts as pedestrian features. Meanwhile, a generative model is used to model the occlusion relationships among pedestrians. The locations where the discriminative model fits the generative model well are thought of as the locations of pedestrians.

Bottom-Up Approach Khan and Shah [15][14] projected the foreground likelihood maps of individual views to a reference view using multi-plane homographies. Areas with heavily overlaid foregrounds are thought of as the locations of pedestrians. However, the foreground projections of different pedestrians may overlap, which leads to false positive detections. Eshel and Moses [8] projected the individual views to the head plane and detected pedestrians at the locations where the intensities projected from different views are pixelwise correlated. Ge and Collins [10] modelled each pedestrian as a cylinder and used Gibbs sampling to find the locations of pedestrians. Utasi and Benedek [22] also used cylinders in the 3D space to model the foreground silhouettes, which was enhanced by pixel-level leg and head features, and determined the pedestrians’ locations by using a 3D Bayesian Marked Point Process model. Xu et al. [24] detected pedestrians in individual views using Faster RCNN [19] and projected the foot points of the bounding boxes of the pedestrians to the ground plane. They clustered the projected foot points to determine the locations of pedestrians in the top view. Hou et al. proposed MVDet [12], an anchor-free end-to-end pedestrian detection network. This system uses ResNet18 [11] as the backbone to extract feature maps from individual views. The feature maps from multiple views are projected to the ground plane and concatenated there. Then a ground-plane classifier predicts the locations of pedestrians. This feature projection method is similar to that proposed by Zhang and Chan [26][27] for multi-camera crowd counting. On the basis of the MVDet framework, Song et al. [21] proposed the SHOT algorithm which projects the feature map of each individual view to multiple parallel planes. The multi-plane feature maps projected from the same view were weighted and summed into one feature map. Such feature maps from the multiple views are concatenated to predict a pedestrian occupancy map on the ground. When the multi-height feature maps were summed into a single feature map, it causes an information loss; whilst such multi-height feature maps are concatenated with no information loss in the 3DROM algorithm, which leads to an improved performance.

In deep-learning based methods, data augmentation methods are widely used to increase the number of training samples and improve the robustness by applying various transformations to existing samples [11][20][16]. One of these methods is to directly apply an image processing operation, such as flipping, folding, rotating, adding noise and Random Erasing, to existing samples. Random Erasing  [28] overwrites each pixel in a randomly selected region of an image with a random colour. This method can be applied to the training of deep-learning based algorithms for image classification, person re-identification and object detection tasks. It improves the robustness of an algorithm to occlusion and reduces the risk of overfitting the samples in the training. In addition, Wang et al. [23] proposed a method for generating samples with occlusion and deformation using adversarial networks. These generated samples can improve the accuracy and robustness of Faster R-CNN in the detection of deformed or occluded objects. However, both methods are currently used in monocular detection only and cannot work well for deep end-to-end multi-view pedestrian detection without considering the geometrical relationship among multiple views.

Let $C$ be the number of the cameras in a multi-view pedestrian dataset. The size of input image $I_{c}$, from camera view $c$ $(c\in[1,C])$, is $H^{c}\times W^{c}$. $(u^{c},v^{c})$ represents an image coordinate in view $c$. The size of the feature map $F_{c}$, extracted from camera view $c$, is $\widetilde{H}^{c}\times\widetilde{W}^{c}$. $H^{t}\times W^{t}$ is the size of the top view image. Assume the area of interest (AOI) on the ground plane is discretized into a grid of $G$ locations. Let $\textbf{X}_{i}$ be the coordinate of the $i$-th location $(i\in[1,G])$. Let $S$ denote the set of the index numbers for the grid locations that have been selected to place the 3D occlusions.

Planar homography is the relationship between a pair of captured images of the same plane. Let u and X be the homogeneous image coordinates of the same point on a plane in camera view $c$ and the top view. They are associated by the homography matrix $\textbf{H}^{c,t}$ for that plane as follow:

A $3\times 4$ projection matrix can be calculated by using the intrinsic and extrinsic parameters of camera $c$: $\textbf{M}=[\textbf{m}_{1},\textbf{m}_{2},\textbf{m}_{3},\textbf{m}_{4}]$. The homography matrix, from the top view $t$ to camera view $c$, for the ground plane is:

The homography matrix, from the top view $t$ to camera view $c$, for the plane parallel to the ground plane and at a height of $h$ can be written as:

where $[\textbf{0}]$ is a $3\times 2$ zero matrix.

Compared with single-view detection, multi-view pedestrian detection requires the use of geometric constraints to establish the correspondence among multiple views. Monocular data augmentation methods, such as flipping, cropping, rotation and Random Erasing, may affect the performance of multi-view detection algorithms, since they violate the homography constraint. Therefore, Algorithm 1 was developed as a 3D data augmentation method for the training of multi-view pedestrian detection algorithms.

The 3D Random Occlusion algorithm is based on the input camera views in the training, as shown in Algorithm 1. The process of 3D Random Occlusion is illustrated in Fig. 2. The ground plane is discretized into a grid of locations. The $i$-th location $(i\in[1,G])$ in the top view is associated with its corresponding location $(u^{c}_{i},v^{c}_{i})$ in camera view $c$ $(c\in[1,C])$ through the ground-plane homography $\textbf{H}^{t,c}_{0}$. A 3D cylinder placed at the $i$th location on the ground plane is back-projected to a filled rectangle $r_{i}^{c}$ sitting at location $(u_{i}^{c},v_{i}^{c})$ in camera view $c$. The rectangle is designed to have the average height $H_{i}^{c}$ and width $W_{i}^{c}$ of the pedestrians standing at the $i$th location. $H_{i}^{c}$ is calculated as follows: the $i$-th location in the top view is projected back to camera view $c$ using the homographies, $\textbf{H}^{t,c}_{0}$ and $\textbf{H}^{t,c}_{h_{a}}$, for the planes at the heights of 0 cm and 180 cm; The vertical distance between the two projected points in view $c$ is $H_{i}^{c}$; the average width $W_{i}^{c}=\alpha H_{i}^{c}$, where $\alpha$ is a constant ratio.

The inputs of Algorithm 1 are the images $I=\{I_{1},I_{2},\cdots,I_{c}\}$ from multiple camera views, the number of occlusions $n$ per frame and the occlusion probability $p$ of each frame to be selected to add 3D random occlusions. The $n$ locations in the top view are selected to generate filled rectangles at the corresponding locations in all the views. To ensure that the occlusions are not too close to each other, the ground distance between each selected location and other cylinder occlusions must be greater than a threshold $d=1$ meter. The selected locations are projected to all the views, by using homographies $\textbf{H}^{t,c}_{0}$ and $\textbf{H}^{t,c}_{h_{a}}$, to generate the filled rectangles with a constant pixel value $\Omega$.

Within the MVDet framework, the feature map $F_{c}$ of view $c$ is projected to the ground plane by using a homography transformation. The feature map in the output of the backbone network does not have the same size as the input image $I_{c}$ of view $c$. However, it is resized to the same size afterwards. Therefore, the projected feature map $F_{h}^{c,t}$ from view $c$ to the top view can be written as:

for a plane parallel to the ground and at a height of $h$.

The feature map on the ground plane is compromised when pedestrians’ feet are occluded or their feet are off the ground. This may affect the model to infer the locations of the pedestrians. In [14], foreground likelihood maps are projected to multiple planes parallel to the ground plane and at different heights, which can significantly reduce detection errors. The foreground likelihood map, which indicates how likely a pixel in an image belongs to foregrounds, is similar to the feature map in MVDet. We assume that each pedestrian occupies a specific location in the top view. The multi-layer projection of the feature maps of multiple views can provide the comprehensive feature information for any pedestrian standing at that location. Compared with the ground-plane projection used in MVDet, the top view CNN is able to infer the locations of pedestrians from a wider range of features.

The multi-layer feature projection is illustrated in Fig. 3(a) and (b). The projected features (or silhouette) of a pedestrian is like the shadow of that pedestrian. When the features of a pedestrian are projected from multiple views to a specific plane, they intersect at the body parts of that pedestrian at the height of that plane. By using multi-plane feature projection, the features across the height of each pedestrian can be utilized. An example of the multi-layer feature projection is shown in Fig. 3(c)-(g).

The projected feature maps and the ground-plane coordinate map are concatenated for the inference of pedestrian locations. The concatenated feature maps are denoted as:

where $M$ is the number of the parallel planes used for feature map projection.

Since the feature maps are projected to the top view with geometric deformation, a layer of DCNv2 [29] is added to the top view CNN to handle the geometric deformation in the projected feature maps. The DCNv2 layer is a complementary component used with the multi-layer projection in 3DROM.

The loss function is the same with that of the MVDet [12]. The network output is an occupancy probability map $\tilde{g}$. A Gaussian kernel $f(\cdot)$ is used to blur the ground-truth pedestrian occupancy map $g$. The loss of the top view $L_{t}$ is the Euclidean distance between them:

The loss function of the single view detection in camera view $c$ is:

where $\tilde{s}_{head}^{c}$ and $\tilde{s}_{foot}^{c}$ are the single-view likelihood maps for heads and feet, respectively; $s_{head}^{c}$ and $s_{foot}^{c}$ are the ground-truth location maps for heads and feet, respectively.

The overall loss for training 3DROM combines the single view loss $L_{single}$ and the top view loss $L_{t}$. It can be written as:

The proposed method has been evaluated on the EPFL WILDTRACK [3][6], MultiviewX [1][12] and EPFL Terrace datasets [2]. These three public video datasets have been widely used to evaluate multi-view pedestrian detection algorithms. Tab. 1 shows the detailed information of these datasets.

The proposed 3DROM method is based on the MVDet framework. Therefore, most of the network parameters were set to the same values as those in MVDet. ResNet-18 was used as the backbone network without using a pre-trained model. The kernel of DCNv2 used in location regression in the top view was set to a size of $2\times 2$. The setup of the input image size, the feature map size, the top view grid size and the number of 3D random occlusions for each dataset are shown in Tab. 1. The 3D random occlusions were added to each frame before the images were input to the backbone in the training.

For the training and testing on all the three datasets, the number of projection layers was set to $M=5$. The feature maps were projected to 5 parallel planes at the heights of 0 cm, 15 cm, 30 cm, 60 cm and 90 cm, respectively. The batch size was set to 1. The occlusion probability $p$ was set to 100%. All experiments were carried out using one RTX-3090 GPU.

The performance of 3DROM on three datasets is demonstrated in the qualitative evaluation. Fig. 4 shows the detection results at frame 3225 of the EPFL Terrace dataset with four camera views. The red rectangle on the ground shows the AOI region. The pedestrians outside the AOI were ignored in the detection and evaluation. The camera positions labelled in the top view are approximate ones. The colour points in the top view represent the detected pedestrians. Meanwhile, the colour of each point in the top view is consistent with the colour of the bounding boxes of the same pedestrian in all the camera views. As can be seen in Fig. 4, the pedestrian in the pink bounding box is completely occluded in C0, partially occluded in C1 and C2, and out of the field of view in C3. The 3DROM method can still infer the location of this pedestrian using limited pedestrian features, which demonstrates its strong detection capability in heavy occlusion.

Fig. 5 shows the detection results at frame 1960 of the EPFL WILDTRACK dataset with seven camera views. The pedestrians stand in a group at the centre of the square and are occluded by each other. The 3DROM algorithm combines the feature information in the multi-view and multi-layer feature projections. These pedestrians are detected correctly by 3DROM.

Fig. 6 shows the detection results at frame 399 of the MultiviewX dataset with six camera views. A large number of pedestrians are standing very close to the border of the AOI in the top view with limited feature information. The use of multi-layer feature projection and 3D Random Occlusion in the training allows the 3DROM algorithm to detect such pedestrians accurately.

The proposed method was evaluated using performance metrics Multiple Object Detection Accuracy (MODA) [13], Multiple Object Detection Precision (MODP) [13], Precision (Prec.) and Recall, which are widely used for multi-view pedestrian detection. The Hungarian algorithm was used to match the detected pedestrians and ground-truth pedestrians. A distance threshold r = 0.5m on the ground plane was used in the matching.

The proposed method was compared with several state-of-the-art deep-learning based methods such as RCNN-2D/3D [24], POM-CNN [9], DeepMCD [7], Deep Occlusion [5], MVDet [12] and SHOT [21], as shown in Tab. 2 in which ”Eval.’ indicates who made the evaluation. The MODA of the 3DROM method is increased to 93.5%, 95.0%, and 94.8% in the evaluation on the WILDTRACK, MultiviewX, and Terrace datasets, respectively. Compared with the baseline algorithm MVDet that uses single-layer projections, the 3DROM increases the MODA by 5.3%, 11.1%, and 7.6%, respectively. Compared with the algorithm SHOT that partly uses multi-layer projections, the 3DROM increases the MODA by 3.3%, 6.7% and 7.7%, respectively. Meanwhile, the 3DROM achieves the best performance in almost all the four performance metrics.

Ablation Study. In order to evaluate the contributions of each component in our model, an ablation study was carried out. The results are shown in Tab. 3, in which M denotes the multi-layer projection and R represents 3D Random Occlusion. As seen from the result, whichever component is added on the baseline MVDet, the performance can have a significant boost in all three datasets. When both components are used in 3DROM, the models are driven to find more robust features across multiple views, and multi-layer projection can provide sufficient features. These two components do not conflict but work better together.

Choice of Projection Layers. To illustrate the benefits of using five-layer feature projections, a validation study was carried out on the Terrace dataset. As reported in Tab. 4, when more than one layer is used in the feature map projection, MODA increases with the number of the projection layers. The experiments show that the feature projection, by using the planes below the waist height (100 cm), leads to better results than that using the planes equidistantly selected between 0 cm and the average pedestrian height 180 cm. This can be interpreted as follows: as can be seen in Fig. 3(a) and (b), in comparison with the ground-plane projection, the feature projection of a pedestrian on a higher plane tends to move towards the underlying camera in the top view, which projects the features, for the pedestrians who are outside of the AOI, into the AOI of the top view. Therefore, by using a projection plane at the pedestrians’ heights, the features, extracted from the distant pedestrians, disturb the pedestrian detection within the AOI. The use of the projection planes below the waist is a good trade-off between the benefits of using multiple planes and the side effects.

Validation of 3D Random Occlusion. To investigate the role of the frequency to use 3D Random Occlusion, we tried different values of the occlusion probability p for using 3D Random Occlusion. As reported in Tab. 5, MODA increases with p and reaches the maximum value when $p=100\%$ in all three datasets. We further compared the 3D Random Occlusion with the related Random Erasing method which was applied to each camera view independently. In this experiment, the optimal settings of Random Erasing [28] proposed by the authors were used. In Tab. 6, The MODA decreases after 3D Random Occlusion is replaced by Random Erasing in all three datasets. This experiment shows the 3D Random Occlusion method can simulate the effect of Random Erasing in 3D space and is specifically designed for multi-view detection.

Fig. 7 shows the validation of the number of 3D random occlusions. When occlusions are too few, the risk of overfitting increases in the training. On the other hand, too many occlusions will cover most pedestrians so that the network cannot learn effective features well. The most appropriate number of occlusions used in training correlates with the average number of pedestrians per frame and the density of pedestrians. Since the WILDTRACK and MultiviewX datasets contain more pedestrians than the Terrace, this number is greater.