KPNet: Towards Minimal Face Detector

In KPNet, we adopt the bottom-up mechanism to perform face detection and alignment simultaneously. Fig. 2 provides an overview of KPNet. Firstly, the potential face scale proposals can be predicted by the fine-grained scale approximation. Secondly, the keypoints can be computed by the scale adaptive soft-argmax with the scalemap and the output heatmap of landmarks response generation. Finally, we apply a simple transformation algorithm (?) to infer the final bounding boxes from the landmarks.

To better detect the facial keypoints, we need to locate the focus regions where existing faces are. The anchor-based mechanism is undoubtedly an appropriate solution, but its’ sophisticated design techniques make it a departure from the simplicity and flexibility of our framework. Inspired by (?; ?) where CNN is capable of approximating the scale information in the low-resolution image, we convert the boxes regression to fine-grained face scale classification for each pixel in a feature map. Different from  (?) where only the existing scales are predicted to select valid layers from the image pyramid, in KPNet, we add additional spatial information to scale approximation.

The scalemap is generated by fine-grained scale approximation that consists of only one convolutional layer with kernel size 3 is used. It is a probability map $M$ with dimension $H^{\prime}\times W^{\prime}\times S$, where $S$ is the predefined number of scales. Given an image $I$ with face boxes $[x,y,h,w]$ where $(x,y)$ means the face center and $h,w$ represent the height and width of the face, the $M$ is firstly initialized to 0 and then we calculate the active channel index $b$ as:

$\displaystyle\begin{split}b=10\times(log_{2}{\frac{max(h,w)\times 2048}{I_{max}}}-5),\end{split}$

(1)

where b is the index from 1 to $S$ and $I_{max}$ represents the max edge of $I$. The minimum detected face size for $I_{max}$=1280 in 720P image is $\bf{20px}$. In Eq. 1, the face size from $2^{5}$ to $2^{11}$ can be mapped into the different channel indexes in $M$. For simplicity, we divide the $[2^{t},2^{t+1}]$, $t\in[5,10]$ into 10 scale bins and thus the total scale number $S=60$. With the computed $b$, the value at coordinate $(\lfloor\frac{x}{N_{s}}\rfloor,\lfloor\frac{y}{N_{s}}\rfloor,b)$ can be defined as:

$\displaystyle\begin{split}M(\lfloor\frac{x}{N_{s}}\rfloor,\lfloor\frac{y}{N_{s}}\rfloor,b)=1,\end{split}$

(2)

where $N_{s}$ means the stride of the network. Encouraged by (?; ?), to alleviate the difficulty of feature learning in the discrete distribution, we introduce the 2D Gaussian function to refine it. Given the radius $r=\lfloor\frac{b}{10}\rfloor$ and the host point $(x_{h},y_{h})=(\lfloor\frac{x}{N_{s}}\rfloor,\lfloor\frac{y}{N_{s}}\rfloor)$, the values of its neighbouring points can be formulated as:

$\displaystyle\begin{split}M(x_{i},y_{i},b)=e^{\frac{(\frac{x_{i}-x_{h}}{r})^{2}+(\frac{y_{i}-y_{h}}{r})^{2}}{2\sigma^{2}}},\end{split}$

(3)

where $(x_{i},y_{i})$ belongs to the neighbour set $\mathcal{N}(x_{h},y_{h},b)$ and $\sigma$ is set to 0.1 in our experiments. During training, the input image is resized with the higher dimension equal to 256 and the loss of the scalemap training is a binary multi-class cross entropy loss:

$\displaystyle\begin{split}L_{scale}=-\frac{1}{|M|}\sum^{|M|}_{i=0}(p_{i}log{\hat{p}_{i}}+(1-p_{i})log(1-\hat{p_{i}})),\end{split}$

(4)

where $p_{i},\hat{p_{i}}$ are the ground truth label and prediction of the i-th pixel in $M$. $|M|$ indicates the pixel number in feature map $M$.

For inference, given a threshold, we select all of the valid coordinates $(x_{v},y_{v},b_{v})$ from $M$ and compute the scale $s_{v}=max(h,w)$ by Eq. 1 based on its channel index $b_{v}$. Finally, the scale proposal $[x_{v},y_{v},s_{v},s_{v}]$ can be obtained.

As shown in (?), heatmap regression models can achieve better performance than coordinate regression models do. With this conclusion in mind, instead of regressing the keypoint coordinates, we detect them from the landmark response map. At the end of the backbone, the landmark response generation is used to generate a response map with dimension $H\times W\times K$ where K is the number of facial keypoints. It only consists of one convolutional layer with kernel size 3. Instead of the argmax function, which is not differentiable, breaking the learning chain on neural networks (?), we propose the $scale$ $adaptive$ $soft$-$argmax$ operator which keeps the properties of specialized part detectors while being fully differentiable. Different from the usage in (?) where it applies to the global response map cooperated with top-down methods, the proposed $scale$ $adaptive$ $soft$-$argmax$ is performed on the scale aware locations cooperated with the bottom-up pipeline.

Given a scale proposal $\mathcal{S}$$=$$[x_{1},y_{1},x_{2},y_{2}]$ where $x_{1},y_{1},x_{2},y_{2}$ mean the top left and bottom right corner, we define the Softmax operation on it $h\in\mathbb{R}^{H\times W\times K}$ as:

$$\Phi(h_{i,j,c})=\left\{\begin{aligned} \frac{e^{h_{i,j,c}}}{\sum_{m=x_{1}}^{x_{2}}\sum_{n=y_{1}}^{y_{2}}e^{h_{m,n,c}}}&,&(i,j)\in\mathcal{S},\\ 0&,&others.\end{aligned}\right.$$

(5)

where $h_{i,j,c}$ is the value of heat map $h$ at location $(i,j)$ of channel $c$. The coordinates of the landmarks $P_{c}=(\Psi_{c,x},\Psi_{c,y})$ corresponding to the $\mathcal{S}$ are given by:

$\displaystyle\begin{split}\Psi_{c,x}=\sum_{i=1}^{W}\sum_{j=1}^{H}\frac{\mathbb{P}(i-x_{1},w)}{w}\Phi(h_{i,j,c})\\ \Psi_{c,y}=\sum_{i=1}^{W}\sum_{j=1}^{H}\frac{\mathbb{P}(j-y_{1},h)}{h}\Phi(h_{i,j,c}),\end{split}$

(6)

where $w,h$ indicate the width and height of $\mathcal{S}$. $\mathbb{P}(x,y)$ returns $x$ when $0\leq x\leq y$ and 0 otherwise.

For keypoints regression based on each $h$, we adopt the $L_{keypoint}$ loss function as follows:

$\displaystyle\begin{split}L_{keypoint}=\frac{1}{2K}\sum_{c=1}^{K}\left\|\Psi_{c,*}-\mathcal{G}_{c,*}\right\|^{2}_{2},\end{split}$

(7)

where $\mathcal{G}_{c,*}$ means the ground truth keypoints. After obtaining the facial keypoints, the face boxes can be inferred by it conforming to the definition of us. Define the probability of a scale proposal as $P_{s}$, the score of the corresponding face box is formulated as:

$\displaystyle\begin{split}P=P_{s}+\sum_{c=1}^{3}max(\Phi(h_{*,*,c})),\end{split}$

(8)

where $max(\cdot)$ means the maximum operation on the spatial resolution and c from 1 to 3 represents the channels corresponding to left eye, right eye, and nose. We empirically utilize the keypoints information to weaken some false positive scale proposals.

We use two different CNN architectures as the backbones of KPNet, respectively. One is the robust stacked hourglass network and the other is DRNet with faster inference speed. The hourglass network only consists of one stacked hourglass and the channel number is reduced to 64. The first convolution layer with kernel size $7\times 7$, stride 2 is replaced by two small convolution layers without BN and ReLU between them. Kernel size $3\times 3$ with stride 2 and kernel size $3\times 3$ with stride 1 are assigned to them, respectively. Furthermore, we upsample the spatial resolution by a factor of 2 (using nearest neighbor upsampling for simplicity) at the end of the hourglass to reduce the bias caused by heavy down-sample operations. Finally, the total stride of the hourglass is 2 and the parameter is $\sim 1.04M$. Even though the network has become very lightweight after these specific modifications, abundant operations on large resolution feature maps still limit the inference speed of the network.

Following the principle of reducing the redundant operation on high-resolution feature maps, we design a simple and fast De-redundancy Net (DRNet). It can achieve $\sim 5\times$ faster inference speed than the former hourglass with the same number of parameters. The details of DRNet are shown in Fig 3. DRNet is a simple and lightweight network with only 11 layers. Given an input image, we first reduce it $4\times$ via a $3\times 3$ convolution layer with stride 2 and a $2\times 2$ max pooling layer with stride 2. Then some residual blocks are followed and two nearest neighbor upsampling layers with factor 2 are used to upsample the spatial resolution. The total stride of DRNet is 2 and the large-span skip layer enables the network to retain as much input information as possible which makes it comfortable for low-resolution input. The lightweight structure without redundant operations on high-resolution feature maps ensures that it can achieve the faster inference speed.

All of the joint face detection and alignment algorithms (?; ?) first generate the face boxes and then predict the corresponding landmarks. Limited by face detection accuracy which heavily relies on the tricky design of anchor boxes and is easily influenced by the ambiguous definition of bounding boxes, it’s hard to achieve better performance with faster speed. Compared with these top-down pipelines, KPNet has the following advantages. First of all, KPNet adopts the bottom-up mechanism that first locates the landmarks with the unambiguous definition, and then the face boxes can be inferred from keypoints. The precise definition of landmarks compared with face boxes allows it to easily achieve higher performance. Secondly, KPNet skips anchor designing and thusly is flexible to deploy this network without complicated design skills. Finally, different from the most face detection methods (?; ?), KPNet does not depend on high-resolution input. Through the combination of low-resolution input and lightweight network, it can achieve SOTA performance on both of the generic face detection ($>$20px) and alignment with fast inference speed (offline application with $\sim 1000$fps).

We implement KPNet in PyTorch. Both of the hourglass and DRNet are randomly initialized under the default setting of PyTorch without pretraining on any external dataset. During training, we set the input resolution of the network to $256\times 256$, which leads to an output resolution of $128\times 128$. For training on generic face detection, we adopt the training set the same as (?) and none of the data augmentations is performed. For joint face detection and alignment, $K$ is set to 5 representing the left eye, right eye, nose, left corner of the mouth and right corner of the mouth. We joint optimize the loss function $L_{scale}$ and the $L_{keypoint}$ with lossweight 1:1 via SGD. Due to the huge pixels in scalemap $M\in\mathbb{R}^{128\times 128\times 60}$, the weight of $L_{scale}$ is set to 10000 for faster convergence. Benefiting from the low-resolution input and lightweight backbone, we use a batch size of 128 and train the network on 4 GTX 1080Ti GPUs. For training on AFLW, because of the missing annotation of some face boxes in the training set, we only train the $L_{keypoint}$ for the annotated facial landmarks. $K$ is set to 19 to correspond to the annotated facial landmarks in AFLW. We adopt the data augmentation strategy the same as (?) for preventing over-fitting. We train the network for 150k iterations with a learning rate warmup strategy. The learning rate is linearly increased to 0.01 from 0.00001 in the first 50k iterations and we reduce it to 0.001 for the last 50k iterations.

At the inference stage, we first generate the scale proposals through the predefined threshold from scalemap, and then compute the corresponding keypoints via then scale adaptive soft-argmax according to Eq. 5 and Eq. 6. Finally, NMS with IOU 0.6 is adopted on the face boxes inferred from these keypoints.

We evaluate KPNet on the generic face detection benchmarks FDDB (?), AFW, MALF, and face alignment benchmark AFLW (?). We adopt the relabeled version provided by (?) where some missing faces are re-annotated. We follow (?) to adopt the AFLW-Full in our experiments where 20,000 and 4,386 images are used for training and testing, respectively. For face detection, we follow the protocol of (?).

Fine-grained scale approximation. Fine-grained scale approximation is a key component of KPNet. To understand its performance, we directly evaluate its recall on FDDB, AFW, and MALF. Similar to the evaluation metric in former section, we compute the recall of the top 100 scale proposals. Furthermore, we implement the anchor-based detector RSA${}_{base}~{}\leavevmode(\nobreak\hskip 0.0pt{\immediate{\bf?}})$, which is the SOTA algorithm on generic face detection. According to their claimed configuration, we evaluate the recall for comparison with the same protocol. The result is shown in Tab. 1. Hourglass_light is the modified hourglass network in Sec. Backbone architecture The fine-grained scale approximation can achieve a comparable recall to the SOTA anchor-based algorithm without relying on the experience design.

Advantage of scale adaptive soft-argmax operator. We introduce the scale adaptive soft-argmax (SS) to predict the facial keypoints coordinates from the heatmap. In order to better evaluate the superiority of SS over $coordinate$ $regression$ and $argmax$, We conduct different experiments with DRNet. For $coordinate$ $regression$, we replace the SS by a fully connected layer to directly regress the keypoints coordinates. We adopt the global average pooling on the scale proposal $\mathcal{S}$ indicated in Sec. Backbone architecture to convert it to a feature vector with the fixed size. A fully connected layer with output $2K$ is applied to regress the keypoint coordinates where $K$ means the keypoint number and $L2$ loss is used for optimization. For $argmax$, each channel in the specific location $\mathcal{S}$ of $M$ corresponding to a specific keypoint and only the coordinates existing keypoints will be set to 1. The loss function is the same as Eq. 4. In the ablation study, we further conduct FDDB${}_{-90^{\circ}}$, FDDB${}_{90^{\circ}}$ and PFDDB as the additional benchmarks. FDDB${}_{-90^{\circ}}$ and FDDB${}_{90^{\circ}}$ are generated by rotating the FDDB with $-90^{\circ}$ and $90^{\circ}$, respectively. PFDDB is assigned by all of the images from FDDB which containing profile faces (Roll $>$ $30^{\circ}$ or Yaw $>$ $30^{\circ}$ or Pitch $>$ $30^{\circ}$).

The results are shown in Tab. 2. All of the models are trained on normal faces without rotation augmentation. SS performs better than others, even the anchor-based RSA_base with high-resolution input and image pyramid, KPNet can achieve comparable performance, even more, robust in face-rotation scenarios. In actually, detecting face boxes from key points is important for the lightweight network than directly regressing bounding boxes. Key points have less uncertain information which is easier for lightweight models to fit. Besides, the semantic information of the landmarks is fixed, even if the face angle/pose changes. This ensures its robustness to face angle/ pose variance.

Error analysis. KPNet simultaneously outputs fine-grained scale heatmap and landmarks response map. To understand how each part contributes to the final error, we perform an error analysis by replacing the predicted scale proposal with the ground-truth boxes. Furthermore, to evaluate the SS contribution to KPNet, we detect the faces only by scale proposal.

Tab. 3 shows that the proposed SS can effectively improve the quality of scale proposal and provide the more precise facial location, especially the recall at false positive number 1. Replacing the SP by GT box improves the recall by $4\sim 5$%. This suggests that there is still ample room for improvement in both SP and SS.