Where, What, Whether: Multi-modal Learning Meets Pedestrian Detection

Compared with front view, bird view owns the unique advantages: 1) The bird view map is naturally free from occlusion issues. We discover that occlusion, whether it is intra-class or inter-class [30], could be mostly attributed to the single detection view (front view). 2) Pedestrians are more likely to be condensed into a unified feature representation through bird view, namely ’head’. Previous anchor-free method, such as CSP which suffers difficulty to decide an ’exact’ center point during training, the proposed ’head’ prediction on bird view map appends more specific and accurate classification targets for the ”Where” branch.

However, how to obtain the bird view map still remains to be solved, which we treat as an image-to-image generation process in this paper. Due to the lack of real bird view map in most of existing pedestrian datasets, such as Citypersons [34] and Caltech [8], we attempt to introduce synthetic data, which directly captured from 3D games to train a bird view generation model. As the previous practice in  [1], synthetic data such as  [25] [18] faces a huge challenge: the performance would be limited if the model trained on synthetic data is directly used on real-world data. Inspired by above, we design our front-to-bird view generation network by two steps. The first includes a CycleGAN [40] [17], denoted as $G_{A\rightarrow B}$ in Figure 2, to transfer real-world data (Domain_A) to the synthetic images (Domain_B). The following step introduces the other CycleGAN, denoted as $G_{B\rightarrow C}$, to train a bird view generation model over synthetic data captured from the game of GTA5, which consists of a total 50,000 pairs of images including front view and bird view from virtual car. Based on the well-trained Generative Adversarial Network (GAN) [15] $G_{A\rightarrow B}$ and $G_{B\rightarrow C}$, the input front view map $I_{F}$ could be transformed to the bird view map $I_{B}$.

Upon $I_{B}$, a detection head, which is composed of one $3\times 3$ conv and two $1\times 1$ conv layers, is attached behind the feature extractor to predict probability of pedestrians at each location and the corresponding width. With its inherent superiority on occlusion, the bird view map can bypass the limitation of front view images, and thus benefit the generation of robust features. However, it is not well performed yet. On the one hand, bird view map pays for the loss of pedestrian height information, which brings about unsatisfied height prediction results on the single bird view map, and thus derives the following ”What” branch to estimate flexible and accurate scales for different instances. On the other hand, bird view map suffers from false positives caused by suspected objects, such as the top of street lamps vs. pedestrian ’head’, thus the introduced ”Whether” branch is directed against this issue.

As is known to us, the bounding box has four-degree-of-freedom, that is $\left\{x,y,w,h\right\}$. During the process of proposal generating, $\left\{x,y\right\}$ are already known, which are evenly distributed on the feature map, and the corresponding $w$ has been predicted on the bird view. Therefore, the ”What” branch only includes two tasks: 1) What the width is on the front view map. 2) What the corresponding height is.

Subsequently, to address above tasks, we start from a more general situation with the following definition.

Definition: We define $v_{0,0}$ as the front view when the image $I_{F}$ is captured at the front horizontally, and $v_{H,\theta}$ as the view when the image $I_{B}$ is captured with the depression angle $\theta$ at the height of $H$. Specially, if $\theta$ is $90\degree$, the view $v_{H,90\degree}$ looks towards perpendicular to horizontal.

On the above basis, we could build up a set of geometric relations. We suppose the real world 3D coordinates of the pedestrian P under the view of $v_{0,0}$ is $(X,Y,Z)$, in which $Z$ could be also considered as the distance from the camera, that is depth,while the corresponding 2D coordinates on the image $I_{F}$ is $(u,v)$, in which given the camera intrinsic matrix $K$, the projection from $(u,v)\rightarrow(X,Y,Z)$ can be formulated as the following:

$$[X,Y,Z]^{T}=ZK^{-1}[u,v,1]^{T}$$

(1)

Following the same steps, 3D coordinates $(X^{\prime},Y^{\prime},Z^{\prime})$ under the view of $v_{H,\theta}$ can also be transformed to 2D coordinates correspondingly, denoted as $(u^{\prime},v^{\prime})$. All of this, the ultimate goal is to build the relation between $(u,v)$ and $(u^{\prime},v^{\prime})$. After the view transformation, there is a relationship that has already been known, which can be written as:

$$\left\{\begin{aligned} X^{\prime}&=X\\ Y^{\prime}&=(Y-H)cos\theta+Zsin\theta\\ Z^{\prime}&=-(Y-H)sin\theta+Zcos\theta\end{aligned}\right.$$

(2)

Besides, same as Equation 1, $(u^{\prime},v^{\prime})$ can be achieved by:

$$[u^{\prime},v^{\prime},1]^{T}=(Z^{\prime})^{-1}K[X^{\prime},Y^{\prime},Z^{\prime}]^{T}$$

(3)

From Equation 1 to Equation 3, the j-th location $l_{j,f}$ with the coordinates $(u,v)$ on the front view map $I_{F}$ could be related to the corresponding location $l_{i,b}$ on the bird view map $I_{B}$, that is $(u^{\prime},v^{\prime})$. Besides, as discussed in Section 3.1, each location $l_{i,b}$ has been assigned with the probability score of pedestrian location and the corresponding width, described as $\left\{p_{i,b},w_{i,b}\right\}$, which could be radiated to $l_{j,f}$.

Above-mentioned formulations are designed to tackle the first task in ”What” branch: what the width of each location is on the front view map. There remains the other issue: what about the height is. Informally speaking, for the specific task of pedestrian detection, the difference in pedestrian height is, to a great extent, depth difference, in which the scale of an instance in the image is inversely proportional to the distance from the camera [37]. In another word, it means the farther away from the camera, the smaller the scale of the target will be. This fact allows us to model interdependency between depth and scale, and treat the process of proposal generation as an uneven distribution problem of scale.

First, still starting from Equation 1, the pedestrian height $\Delta h$ in one image could be modeled as the distance from head $\textbf{d}_{1}=[u_{1},v_{1}]^{T}$ to foot $\textbf{d}_{2}=[u_{2},v_{2}]^{T}$, formulated as $\Delta h=||\textbf{d}_{1}-\textbf{d}_{2}||_{2}$, while the real height $\Delta H$ in the real world could be mapped to $C\Delta H=Z\Delta h$, in which $C$ is a constant composed of camera intrinsic parameters and $Z$ is the depth we mentioned. As analyzed above, $Z$ (depth) and $\Delta h$ (height) are completely inverse when $C\Delta H$ is the fixed value. However, due to the influence of gender and age in pedestrian height, it is slightly unreasonable to set $\Delta H$ as a fixed value directly. In our height estimates, we assume that the height $\Delta H$ of all pedestrians obeys the uniform distribution and we analyze the error of the assumption. Previous studies of 63,000 European adults have shown that the average height of males and females is 178cm and 165 cm respectively, with a standard deviation of 7cm in both cases [28]. Besides, the distribution of human stature follows a Gaussian distribution for male and female populations [13], denoted as: $H_{male}\sim N(\mu_{1},\sigma^{2})$ and $H_{female}\sim N(\mu_{2},\sigma^{2})$, respectively. Specifically, for each point $l_{j,f}$ in the image $I_{F}$, its pedestrian height $\Delta H$ in real world is a sample from the above distribution, and we utilize the normal distribution function to evaluate the uncertainty of the sampled height $\Delta H$. Take the male distribution $H_{male}$ as an example. if the sampling height is $\hat{H}$, the corresponding uncertainty can be formulated as:

$$\hat{e}=P(|x-\mu_{1}|<|\hat{H}-\mu_{1}|)=2\Phi(\frac{|\hat{H}-\mu_{1}|}{\sigma})-1$$

(4)

in which $\Phi(\cdot)$ is function of standard normal distribution. The above $\hat{e}$ models the estimation uncertainty of $\Delta h$ due to variation of pedestrian height $\Delta H$ in real world, in which $\hat{e}$ goes smaller when $\hat{H}$ gets closer to $\mu_{1}$, and instead goes larger. This estimation uncertainty also servers as the confidence value to re-weight loss function, described in Section 3.4. With the above formulated $w$ and $h$, the corresponding proposal in the location of $(x,y)$ could be generated, and subsequently leads to the following ”Whether” branch.

After the ”Where” and ”What” branch, a large number of proposals have been generated, and the corresponding possibility score $p_{j,b}$, which indicates whether there is an pedestrian instance or not in the bird view map, could also be projected to the front view. However, the single bird view prediction has the problem: the increase of false positives. Therefore, we introduce visual and attribute embedding, namely ”Whether” branch, to alleviate the above problem. This branch is based on two observations: one is that many false positives are mainly with human-like vertical structures, such as railings or tree trunks [33]; another is the fact that pedestrians, which perform walking or standing with up-straight pose, have one specific body structure from top to bottom: Head-Arm-Body-Leg. With this pedestrian-specific structure, we attempt to re-encode both occluded and non-occluded ones into the unified representation to distinguish those false positives with global vertical structure but lacking human-body based partial features.

The component of the proposed ”Whether” is shown in Figure 2. It consists of a variational autoencoder (VAE) [7], which is optimized by $L_{DE}$ to learn a lantent space shared by both visual features and attribute embeddings. More specifically, the embedding network takes the attributes embedding vector as input, and after passing through two fully connected (fc) layers and the Rectified Linear Unit (ReLU), outputs a visual embedding vector, which has the same dimensions as the visual feature vector of each proposal extracted from the encoder, optimizing:

$$L_{DE}=\frac{1}{N}\sum_{i=1}^{N}||E(x_{i})-\varphi(y)||_{2}^{2}$$

(5)

where $N$ is the total number of all proposals, $x_{i}$ is the visual feature of the i-th proposal, $y$ represents the word vector of pedestrian attributes including Head, Arm, Body and Leg, $E(\cdot)$ represents the visual feature encoder and $\varphi(\cdot)$ is the embedding network. The output feature via the decoder network used for the following classification and regression predicts the possibility score of whether there is a pedestrian or not, denoted as $p_{j,f}$, which is combined with $p_{j,b}$ and thus the overall possibility score of the j-th proposal could be formulated as:

$$p_{j}=\frac{1}{2}(p_{j,b}+p_{j,f})$$

(6)

Towards this task, we resort to depth maps derived from a CNN-based monocular depth prediction model [1]. The feature generator consists of FPN [20] with a powerful backbone ResNet-50 [16], and the overall framework is optimized by the following loss function.

In the ”Where” branch, according to Equation 1 to 3, the horizontal and vertical central axes of each ground truth are mapped to the bird view, and thus constitute the bird view ground truth. Same as the loss function in CSP [23], we also formulate possibility and width prediction as a classification and regression task via the cross-entropy and smooth L1 loss, respectively, denoted as $L_{bird}$ and $L_{width}$.

In the ”What” branch, it is actually a geometric transformation without extra conv layers introduced. It is worth noting that in order to guarantee the robustness of the detector, pedestrian height $\Delta H$ in the real world is sampled from a distribution rather than using a fixed height value. Therefore, in order to evaluate the uncertainty of different samples, we introduce $\hat{e}$ in Equation 4 and consequently re-weight it on the following classification and regression loss.

In the ”Whether” branch, features extracted from each proposal are first encoded to a latent space, and subsequently decoded as the final classification and regression features, optimized by the loss $L_{cls}$ and $L_{reg}$. The overall loss can be formulated as:

	$\displaystyle L=$	$\displaystyle\lambda_{1}L_{bird}+\lambda_{2}L_{width}+\lambda_{3}L_{DE}$		(7)
		$\displaystyle+\frac{\lambda_{4}}{N}\sum_{i=1}^{N}(1-\hat{e}_{i})(L_{cls,i}+L_{reg,i})$

where $i$ represents the i-th proposal and $N$ is the total number of all proposals. $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ and $\lambda_{4}$ are the weights for each loss, which are experimentally set as 0.01, 0.1, 0.1 and 1, respectively.

Datasets Citypersons  [34] is a diverse dataset built upon the Cityscapes data, which includes 5000 images (2975 for training, 500 for validation, and 1525 for testing). In a total of 5 000 images, it has $\sim$35k person and $\sim$13k ignore region annotations. And it notices the density of persons are consistent across train/validation/test subsets. The Caltech Dataset  [8] consists of approximately 10 hours of 640x480 30Hz video taken from a vehicle driving through regular traffic in an urban environment. About 250,000 frames with a total of 350,000 bounding boxes and 2300 unique pedestrians were annotated. All of the datasets contain challenging settings, denoted as Heavy occlusion, in which the propotion of visible parts of pedestrians is less than $0.65$.

Implementation We implemented the proposed method in Pytorch with the backbone ResNet50 [16] and the Nvidia GTX1080Ti. We optimize the network using the Stochastic Gradient Descent (SGD) algorithm with 0.9 momentum and 0.0005 weight decay, respectively. For Citypersons, the mini-batch contains 2 images and we train the network for $30k$ iterations with the initial learning rate of $10^{-3}$ and decay it to $10^{-4}$ for another $6k$ iterations. For Caltech, the mini-batch contains 8 images and we train the network for $40k$ iterations with the initial learning rate of $10^{-3}$ and decay it to $10^{-4}$ for another $20k$ iterations.

In this section, we evaluate how each significant component of our network contributes to performance using the Citypersons dataset under the Reasonable and Heavy settings, which are specification for the non-occluded and occluded pedestrian. It is worth noting that to evaluate the performance of each branch, such as results in Table 3, we take each ”W” step as one enhanced component, and replace the corresponding module in baseline (Faster RCNN, Table 3 Line 1). For example, to get the result of ”Where”, the process of proposal generation in RPN is replaced by ’head’ and width prediction in the bird view map together with fixed aspect ratio. Or to evaluate the ”What” step, we replace the fixed aspect ratio with the depth based scale estimation. The same is true for experiments in Table 4.

Why is the bird view? Occlusion is one of the great challenges entrenched in pedestrian detection, which is ultimately caused by isolated view point, especially the front view. However, we could actually discover that in the real 3D world, even if a pedestrian is occluded from the initial view, bird view still performs free from occlusion issues. Besides, pedestrian instances are condensed to a point on the bird view, namely ”head”, which can be seamlessly connected with the anchor-free detectors. Inspired by above, experiments are conducted and comparisons are reported in Table 3. The method combined with ”Where” branch achieves 23.3$\%MR^{-2}$ on Heavy occlusion subset, which is an absolute 37.3-point improvement over our baseline 60.6%. When the ”Where” branch is added to CSP, as is shown in Table 4, the performance of CSP is boosted by a large margin from 49.3$\%MR^{-2}$ to 24.6$\%MR^{-2}$. It is in accordance with our intuition that ”Where” branch is specially designed for the long-standing occlusion problem. In addition, ablations are carried out in Table 1 and 5 to investigate whether the difference in generation of bird view map will influence the results. Comparisons reported in Table 1 show that the view $v_{\theta,H}$ under $\theta=60\degree$ and $H=20$ meters achieves the best performance, and we believe that more detailed exploration of the parameters $\theta$ and $H$ can further improve the performance, but it is not in the scope of this work. In Table 5, the method without domain transfer directly generate bird view images from the real-world data suffering a great decline on both Reasonable and Heavy subsets, which demonstrates the effectiveness of the proposed $G_{A\rightarrow B}$.

Why is the depth? Depth is a useful auxiliary information, however, it has not been fully identified and utilized in pedestrian detection. One is that depth implies scale cues, especially for pedestrians, which share strong intra-class similarity. The other is that depth plays a connecting role in our framework. Depth, that is ”What” branch, projects detection results from ”Where” branch to the front view, and on the basis of interdependency between depth and scale, generating reasonable proposals for the following ”Whether” branch. As shown in Table 3 and 4, compared with the baseline or the bare CSP, the proposed ”What” branch goes a step further and achieves superior performance on both subsets. It is still worth noting that ”What” branch is just a kind of matrix transformation without any conv or fc layers introduced, which preserves the effectiveness of the detector while ensuring better performance. To further investigate the effect, we also conduct experiments including a prediction network to estimate pedestrian height, as reported in Table 2, in which the single $w_{b}$ means the detector combines width prediction on bird view with height estimation by depth-scale relations, while $w_{b}+h_{f}$ combines both width and height prediction on bird view and front view, respectively. It can be observed that other methods can achieve comparable but suboptimal results to $w_{b}$ prediction. This result may be attributed to the accurate and consistent height estimation with less noise during training, which has also been proved in Figure 3. Proposals generated by the proposed method shares better iou performance, in which the proportion of $IoU>0.5$ is improved greatly.

Why is the corpus? A natural question is, for occlusion, is the bird view enough? The bird view map can effectively provide the locations of all pedestrians. However, due to ambiguous pedestrian features, suspected objects, such as railings, which are similar to pedestrian ’head’ in the bird view, could also be falsely classified as the pedestrian. In order to tackle this problem, we introduce the ”Whether” branch, which falls into the proposal detection fashion on the basis of the front view map. The difficulty lies in proposal classification is that features of occluded pedestrians usually perform incomplete compared with those of non-occluded instances, that is to say, these two types have different feature distribution. Therefore, we re-encode proposals into a unified corpus space, which with fixed dimension and clear semantic information generates robust relations between visual and semantic cues and thus benefits robust feature representation. Results are reported in Table 3 and 4. It can be seen that although ”What” and ”Whether” branch both focus on occluded pedestrian detection, their detection uses different modal input, which maximises the complementary information of different views and thus achieve the best performance finally.

Performance compared with state-of-the-art methods on Citypersons Reasonable, Partial and Heavy dataset is shown in Table 6, while the result on Caltech is reported in Table 7. It can be observed that 1) W³Net leads a new state-of-the-art result of 9.3% (Citypersons) and 3.82% (Caltech) on both Reasonable subsets.It is note worthy that at the same time, the proposed method also achieves promising performance on occluded instances. In particular,when evaluating heavy occlusion subset on Citypersons, our model produces18.7% while prior arts range from 49%-56%. 2) Result on Far subset of Caltech outperforms previous advanced detectors, such as TLL [27] by 16.98$\%$ which is specially designed for small-scale targets, demonstrates the superiority on scale variation. 3) Despite introducing multi-modal data, the careful discovery of depth relation actually reduces computational cost without extra conv or fc layers, and thus the inference time on one GTX 1080Ti with the 1x. image scale is still in line with the state-of-the-arts.