Heatmap Regression via Randomized Rounding

Coordinate regression has been widely and successfully used in semantic landmark localization under constrained settings, where it usually relies on simple yet effective features [30, 31, 32, 33]. To improve the performance of coordinate regression for semantic landmark localization in the wild, several methods have been proposed by using cascade refinement [1, 34, 35, 36, 37, 38], parametric/non-parametric shape models [39, 29, 36, 40], multi-task learning [41, 42, 43], and novel loss functions [44, 45].

The success of deep learning has prompted the use of heatmap regression for semantic landmark localization, especially for robust and accurate facial landmark localization [2, 46, 47, 23] and human pose estimation [19, 48, 6, 49, 50, 51, 52]. Existing heatmap regression methods either rely on large input images or empirical compensations during inference to mitigate the problem of quantization error [6, 53, 54, 22]. For example, a simple yet effective compensation method known as “shift a quarter to the second maximum activation point” has been widely used in many state-of-the-art heatmap regression methods [6, 55, 22].

Several methods have been developed to address the problem of quantization error in three aspects: 1) jointly predicting the heatmap and the offset in a multi-task manner [56]; 2) encoding and decoding the fractional part of numerical coordinates via a modulated 2D Gaussian distribution [24, 26]; and 3) exploring differentiable transformations between the heatmap and the numerical coordinates [57, 20, 58, 59]. Specifically, [24] generates the fractional part sensitive ground truth heatmap for video-based face alignment, which is known as fractional heatmap regression. Under the assumption that the predicted heatmap follows a 2D Gaussian distribution, [26] decodes the fractional part of numerical coordinates from the modulated predicted heatmap. The soft-argmax operation is differentiable [60, 61, 62, 59], and has been intensively explored in human pose estimation [57, 20].

Heatmap regression for semantic landmark localization usually contains two key components: 1) heatmap prediction; and 2) transformation between the heatmap and the numerical coordinates. The quantization system in heatmap regression is a combination of the encode and decode operations. During training, when the ground truth numerical coordinates $\boldsymbol{x}_{i}^{g}$ are floating-point numbers, we then need to calculate a specific Gaussian kernel matrix using (3) for each landmark, since different numerical coordinates usually have different fractional parts. As a result, it will significantly increase the training loads of the heatmap regression model. For example, given $98$ landmarks per face image, the kernel size $11\times 11$, and a mini-batch of $16$ training samples, we then have to run (3) for $98\times 16\times 11\times 11=189,728$ times in each training iteration. To address this issue, existing heatmap regression methods usually first quantize numerical coordinates into integers, where a standard kernel matrix can then be shared for efficient ground truth heatmap generation [6, 2, 55, 23]. However, the above-mentioned existing heatmap regression methods usually suffer from the inherent drawback of failing to encode the fractional part of numerical coordinates. Therefore, how to efficiently encode the fractional information in numerical coordinates still remains challenging. Furthermore, during the inference stage, the predicted numerical coordinates $\boldsymbol{x}_{i}^{p}$ obtained by a decode operation in (2) are also integers. As a result, typical heatmap regression methods usually fail to efficiently address the fractional part of the numerical coordinate during both training and inference, resulting in localization error.

To analyze the localization error caused by the quantization system in heatmap regression, we formulate the localization error as the sum of heatmap error and quantization error as follows:

where $\boldsymbol{x}_{i}^{opt}$ indicates the numerical coordinate decoded from the optimal predicted heatmap. Generally, the heatmap error corresponds to the error in heatmap prediction, i.e., $\|\boldsymbol{h}_{i}^{p}-\boldsymbol{h}_{i}^{g}\|_{2}$, and the quantization error indicates the error caused by both the encode and decode operations. If there is no heatmap error, the localization error then all originates from the error of the quantization system, i.e.,

The generalizability of deep neural networks for heatmap prediction, i.e., the heatmap error, is beyond the scope of this paper. We do not consider the heatmap error during the analysis of quantization error in this paper.

To obtain integer coordinates for the generation of the ground truth heatmap, typical integer quantization operations such as floor, round, and ceil have been widely used in previous heatmap regression methods. To unify the quantization error induced by different integer operations, we first introduce a unified integer quantization operation as follows. Given a downsampling stride $s>1$ and a threshold $t\in[0,1]$, the coordinate $x\in\mathbb{N}$ then can be quantized according to its fractional part $\epsilon=x/s-\lfloor x/s\rfloor$, i.e.,

That is, for integer quantization operations floor, round, and ceil, we have $t=1.0$, $t=0.5$, and $t=0$, respectively. Furthermore, when the downsampling stride $s>1$, the decode operation in (2) then becomes

A vanilla quantization system for heatmap regression can then be formed by the encode operation in (10) and the decode operation in (11). When applied to a vector or a matrix, the integer quantization operation defined in (10) is an element-wise operation.

In this subsection, we first correct the bias in a vanilla quantization system to form an unbiased vanilla quantization system. With the unbiased quantization system, we then provide a tight upper bound on the quantization error for vanilla heatmap regression.

Let $\epsilon_{x}$ denote the fractional part of $x_{i}^{g}/s$, and $\epsilon_{y}$ denote the fractional part of $y_{i}^{g}/s$. Given the downsampling stride of the heatmap $s>1$, we then have

Given the assumption of a “perfect” heatmap prediction model or no heatmap error, i.e., $\boldsymbol{h}_{i}^{p}(\boldsymbol{x})=\boldsymbol{h}_{i}^{g}(\boldsymbol{x})$, we then have the predicted numerical coordinates

If data samples satisfy the i.i.d. assumption and the fractional parts $\epsilon_{x},\epsilon_{y}\in\mathbb{U}(0,1)$, the bias of $\boldsymbol{x}_{i}^{p}$ as an estimator of $\boldsymbol{x}_{i}^{g}$ can then be evaluated as

Considering that $x_{i}^{g},y_{i}^{g}$ are independent variables, we thus have the quantization bias in the vanilla quantization system as follows:

Therefore, only the encode operation in (10), i.e., the round operation, is unbiased. Furthermore, given $\forall t~{}\in~{}[0,1]$ for the encode operation in (10), we can correct the bias of the encode operation with a shift on the decode operation, i.e.,

In vanilla heatmap regression, each numerical coordinate corresponds to a single activation point in the heatmap, while the indices of the activation point are all integers. As a result, the fractional part of the numerical coordinate is usually ignored during the encode process, making it an inherent drawback of heatmap regression for sub-pixel localization. To retain the fractional information when using heatmap representations, we utilize multiple activation points around the ground truth activation point. Inspired by the randomized rounding method [27], we address the quantization error in vanilla heatmap regression by using a probabilistic approach. Specifically, we encode the fractional part of the numerical coordinate to different activation points with different activation probabilities. An intuitive example is shown in Fig. 3.

We describe the proposed quantization system as follows. Given the ground truth numerical coordinate $\boldsymbol{x}_{i}^{g}=(x_{i}^{g},y_{i}^{g})$ and a downsampling stride of the heatmap $s>1$, the ground truth activation point in the heatmap is $(x_{i}^{g}/s,y_{i}^{g}/s)$, which are usually floating-point numbers, and we are unable to find the corresponding pixel in the heatmap. If we ignore the fractional part $(\epsilon_{x},\epsilon_{y})$ using a typical integer quantization operation, e.g., round, the ground truth activation point will be approximated by one of the activation points around the ground truth activation point, i.e., $\left(\lfloor x_{i}^{g}/s\rfloor,\lfloor y_{i}^{g}/s\rfloor\right)$, $\left(\lfloor x_{i}^{g}/s\rfloor+1,\lfloor y_{i}^{g}/s\rfloor\right)$, $\left(\lfloor x_{i}^{g}/s\rfloor,\lfloor y_{i}^{g}/s\rfloor+1\right)$, and $\left(\lfloor x_{i}^{g}/s\rfloor+1,\lfloor y_{i}^{g}/s\rfloor+1\right)$. However, the above process is not invertible. To address this, we randomly assign the ground truth activation point to one of the alternative activation points around the ground truth activation point, and the activation probability is determined by the fractional part of the ground truth activation point as follows:

To achieve the encode scheme in (14) in conjunction with current minibatch stochastic gradient descent training algorithms for deep learning models, we introduce a new integer quantization operation via randomized rounding, i.e., random-round:

Given the encode operation in (15), if we do not consider the heatmap error, we then have the activation probability at $\boldsymbol{x}$:

As a result, the fractional part of the ground truth numerical coordinate $(\epsilon_{x},\epsilon_{y})$ can be reconstructed from the predicted heatmap via the activation probabilities of all activation points, i.e.,

where $\mathcal{X}_{i}^{g}$ indicates the set of activation points around the ground truth activation point, i.e.,

The fractional information of the numerical coordinate $(\epsilon_{x},\epsilon_{y})$ is well-captured by the randomized rounding operation, allowing us to reconstruct the ground truth numerical coordinate $\boldsymbol{x}_{i}^{g}$ without the quantization error. However, during the inference phase, the ground truth numerical coordinate $\boldsymbol{x}_{i}^{g}$ is unavailable and heatmap error always exists in practice, making it difficult to identify the proper set of ground truth activation points $\mathcal{X}_{i}^{g}$. In this section, we describe a way to form a set of alternative activation points in practice.

We introduce two activation point selection methods as follows. The first solution is to estimate all activation points via the points around the maximum activation point. As shown in Fig. 4, given the maximum activation point, we then have four different sets of alternative activation points, $\mathcal{X}_{i}^{g_{1}},\mathcal{X}_{i}^{g_{2}},\mathcal{X}_{i}^{g_{3}},\text{and}~{}\mathcal{X}_{i}^{g_{4}}$. Therefore, given the predicted heatmap in practice, we then take a risk of choosing an incorrect set of alternative activation points. To find a robust set of alternative activation points, we may use all nine activation points around the maximum activation point, i.e.,

Another solution of alternative activation points $\mathcal{X}_{i}^{g}$ is to generalize the argmax operation with the argtopk operation, i.e., we decode the predicted heatmap $\boldsymbol{h}_{i}^{p}$ to obtain the numerical coordinate $\boldsymbol{x}_{i}^{p}$ according to the top $k$ largest activation points,

If there is no heatmap error, the two alternative activation points solutions presented above, i.e., the alternative activation points in (18) and (19), are equal to each other when using the decode operation in (17). Specifically, we find that the activation points in (19) achieve comparable performance to the activation points in (20) when $k=9$. For simplicity, unless otherwise mentioned, we use the set of alternative activation points defined by (20) in this paper. Furthermore, when we take the heatmap error into consideration, the values of different $k$ then forms a trade-off on the selection of activation points, i.e., a larger $k$ will be robust to activation point selection whilst also increasing the risk of noise from the heatmap error. See more discussion in Section 4.5 and the experimental results in Section 5.5.

In this subsection, we provide some insights into the proposed quantization system with respect to: 1) the influence of human annotators on the proposed quantization system in practice; and 2) the underlying explanation behind the widely used empirical compensation method “shift a quarter to the second maximum activation point”.

Unbiased Annotation. We assume the “ground truth numerical coordinates” are always accurate, while the ground truth numerical coordinates are usually labelled by human annotators at the risk of the annotation bias. Given an input image, the ground truth numerical coordinates $\boldsymbol{x}_{i}^{g}$ can be obtained by clicking a specific pixel in the image, which is a simple but effective annotation pipeline provided by most image annotation tools. For sub-pixel numerical coordinates, especially in low-resolution input images, the annotators may click either one of all possible pixels around the ground truth numerical coordinates due to human visual uncertainty. As shown in Fig. 5, clicking any one of the four possible pixels causes annotation error, which corresponds to the fractional part of the ground truth numerical coordinate $(\epsilon_{x}^{\prime},\epsilon_{y}^{\prime})=\left(x_{i}^{g}-\lfloor x_{i}^{g}\rfloor,~{}y_{i}^{g}-\lfloor y_{i}^{g}\rfloor\right)$. Given enough data samples, if the annotators click the pixel according to the following distribution, i.e.,

the fractional part then can be well captured by the heatmap regression model and we refer to it as an unbiased annotation.

If we take the downsampling stride into consideration, $(\epsilon_{x},\epsilon_{y})$ is then a joint result of both the downsampling of the heatmap and the annotation process, i.e.,

On the one hand, if the heatmap regression model uses a low input resolution (or a large downsampling stride $s\gg 1$), the fractional part $(\epsilon_{x},\epsilon_{y})$ then mainly comes from the downsampling of the heatmap; on the other hand, if the heatmap regression model uses a high input resolution, the annotation process will also have a significant influence on the heatmap regression. Therefore, when using a high input resolution model in practice, a diverse set of human annotators help reduce the bias in annotation process.

Empirical Compensation. “Shift a quarter to the second maximum activation point” has become an effective and widely used empirical compensation method for heatmap regression [6, 55, 22], but it still lacks a proper explanation. We thus provide an intuitive explanation according to the proposed quantization system. The proposed quantization system encodes the ground truth numerical coordinates into multiple activation points, and the activation probability of each activation point is decided by the fractional part, i.e., the activation probability indicates the distance between the activation point and the ground truth activation point. Therefore, the ground truth activation point is closer to the $i$-th maximum activation point than the $(i+1)$-th maximum activation point. We demonstrate the averaged activation probabilities for the top $k$ activation points on the WFLW dataset in Table I.

We find that the marginal improvement decreases as the number of activation points increases, i.e., the second maximum activation points provides the maximum improvement to the reconstruction of the fractional part. This observation partially explains the effectiveness of the compensation method “shift a quarter to the second maximum activation point”, which can be seen as a special case of the proposed method (20) with $k=2$.

Furthermore, the proposed quantization system shares the same motivation with the bilinear interpolation. Specifically, the bilinear interpolation usually aims to find the value of the unknown function $f(x,y)$ given its neighbors $f(x_{1},y_{1})$, $f(x_{1},y_{2})$, $f(x_{2},y_{1})$, and $f(x_{2},y_{2})$. For the proposed quantization system, we have $f(x,y)=(x,y)$, which indicates the location of landmarks. Specifically, if there is no heatmap error, we then have $x_{1}=\lfloor x_{i}^{g}/s\rfloor$, $x_{2}=\lfloor x_{i}^{g}/s\rfloor+1$, $y_{1}=\lfloor y_{i}^{g}/s\rfloor$, and $y_{2}=\lfloor y_{i}^{g}/s\rfloor+1$. If we take heatmap error into consideration, the ground truth activation points are usually unknown. Therefore, the number of alternative activation points also controls the trade-off between the robustness of the quantization system and the risk of noise from the heatmap error (Also see details in Section 4.4).

We use four widely used facial landmark detection datasets:

• •

  WFLW [64]. WFLW contains $10,000$ face images, including $7,500$ training images and $2,500$ testing images, with $98$ manually annotated facial landmarks. All face images are selected from the WIDER Face dataset [74], which contains face images with large variations in scale, expression, pose, and occlusion.

• •

  300W [75]. 300W contains $3,148$ training images, including $337$ images from AFW [30], $2,000$ images from the training set of HELEN [76], and $811$ images from the training set of LFPW [39]. For testing, there are four different settings: 1) common: $554$ images, including $330$ and $224$ images from the testsets of HELEN and LFPW, respectively; 2) challenge: $135$ images from IBUG; 3) full: $689$ images as a combination of common and challenge; and 4) private: $600$ indoor/outdoor images. All images are manually annotated with $68$ facial landmarks.

• •

  COFW [34]. COFW contains $1,852$ images, including $1,345$ training and $507$ testing images. All images are manually annotated with $29$ facial landmarks.

• •

  AFLW [77]. AFLW contains $24,386$ face images, including $20,000$ images for training and $4,836$ images for testing. For testing, there are two settings: 1) full: all $4,836$ images for testing; and 2) front: $1,314$ frontal images selected from the full set. All images are manually annotated with $21$ facial landmarks. For fair comparison, we use $19$ facial landmarks, i.e., the landmarks on two ears are ignored.

We use the normalized mean error (NME) as the evaluation metric in this paper, i.e.,

where $d$ indicates the normalization distance. For fair comparison, we report the performances on WFLW, 300W, and COFW using two the normalization methods, inter-pupil distance (the distance between the eye centers) and inter-ocular distance (the distance between the outer eye corners). We report the performance on AFLW using the size of the face bounding box as the normalization distance, i.e., the normalization distance can be evaluated by $d=\sqrt{w*h}$, where $w$ and $h$ indicate the width and height of the face bounding box, respectively.

We implement the proposed heatmap regression method for facial landmark detection using PyTorch [78]. Following the practice in [23], we use HRNet [22] as our backbone network, which is an efficient counterpart of ResNet [79], U-Net [80], and Hourglass [6] for semantic landmark localization. Unless otherwise mentioned, we use HRNet-W18 as the backbone network in our experiments. All face images are cropped and resized to $256\times 256$ pixels and the downsampling stride of the feature map is $4$ pixels. For training, we perform widely-used data augmentation for facial landmark detection as follows We horizontally flip all training images with probability $0.5$ and randomly change the brightness ($\pm 0.125$), contrast ($\pm 0.5$), and saturation ($\pm 0.5$) of each image. We then randomly rotate the image ($\pm 30^{\circ}$), rescale the image ($\pm 0.25$), and translate the image ($\pm 16$ pixels). We also randomly erase a rectangular region in the training image [81]. All our models are initialized from the weights pretrained on ImageNet [82]. We use the Adam optimizer [83] with batch size $16$. The learning rate starts from $0.001$ and is divided by $10$ for every $60$ epochs, with $150$ training epochs in total. During the testing phase, we horizontally flip testing images as the data augmentation and average the predictions.

To demonstrate the effectiveness of the proposed method, we compare it with recent state-of-the-art facial landmark detection methods as follows. As shown in Table II, the proposed method outperforms recent state-of-the-art methods on the most challenging dataset, WFLW, with a clear margin for all different settings. For the 300W dataset, we try to report the performances under different settings for fair comparison. As shown in Table III, the proposed method achieves comparable performances for all different settings. Specifically, LAB [64] uses the boundary information as the auxiliary supervision; compared to Wing [45], which uses the coordinate regression for semantic landmark localization, the heatmap-based methods usually achieve better performance on the challenge subset. In Table IV, we see that the proposed method outperforms recent state-of-the-art methods with a clear margin on COFW dataset. AFLW captures a wide range of different face poses, including both frontal faces and non-frontal faces. As shown in Table V, the proposed method achieves consistent improvements for both frontal faces and non-frontal faces, suggesting robustness across different face poses.

To better understand the proposed quantization system in different settings, we perform ablation studies the most challenging dataset, WFLW [64].

The influence of different input resolutions. Heatmap regression models use a fixed input resolution, e.g., $256\times 256$ pixels, but training and testing images usually represent a wide range of resolutions, e.g., most faces in the WFLW dataset have an inter-ocular distance of between $30$ and $120$ pixels. Therefore, we compare the proposed method with the baseline using an input resolution from $64\times 64$ pixels to $512\times 512$ pixels, i.e., a heatmap resolution from $16\times 16$ pixels to $128\times 128$ pixels. In Fig. 6, the proposed method significantly improves heatmap regression performance when using a low input resolution. The increasing number of high-resolution images/videos in real-world applications is a challenge with respect to the computational cost and device memory needed to overcome the problem of sub-pixel localization by increasing the input resolution of deep learning-based heatmap regression models. For example, in the film industry, it has sometimes become necessary to swap the appearance of a target actor and a source actor to generate higher fidelity video frames in visual effects, especially when an actor is unavailable for some scenes [84]. The manipulation of actor faces in video frames relies on accurate localization of different facial landmarks and is performed at megapixel resolution, inducing a huge computational cost for extensive frame-by-frame animation. Therefore, instead of using high-resolution input images, our proposed method delivers another efficient solution for dealing with accurate semantic landmark localization.

The influence of different numbers of alternative activation points. In the proposed quantization system, the activation probability indicates the distance between activation point to the ground truth activation point $(x_{i}^{g}/s,y_{i}^{g}/s)$. If there is no heatmap error, the alternative activation points in (20) then give the same result as in (18). If the heatmap error cannot be ignored, there will be a trade-off on the number of alternative activation points: 1) a small $k$ increases the risk of missing the ground truth alternative activation points; 2) a large $k$ introduces the noise from irrelevant activation points, especially for large heatmap error. We demonstrate the performance of the proposed method by using different numbers of alternative activation points in Table VI. Specifically, we see that 1) when using a high input resolution, the best performance is achieved with a relatively large $k$; and 2) the performance is smooth near the optimal number of alternative activation points, making it easy to find a proper $k$ for validation data. As introduced in Section 3, binary heatmaps can be seen as a special case of Gaussian heatmaps with standard deviation $\sigma=0$. Considering that Gaussian heatmaps have been widely used in semantic landmark localization applications, we generalize the proposed quantization system to Gaussian heatmaps and demonstrate the influence of different numbers of alternative activation points in Table VII. Specifically, we see that 1) when applying the proposed quantization system to the model using Gaussian heatmap, it achieves comparable performance to the model using binary heatmap; and 2) the optimal number of alternative activation points increases with the standard deviation $\sigma$.

The influence of different “bounding box” annotation policies. For facial landmark detection, a reference bounding box is required to indicate the position of the facial area. However, there is a performance gap when using different reference bounding boxes [21]. A comparison between two widely used “bounding box” annotation policies is shown in Fig. 7, and we introduce two different bounding box annotation policies as follows:

• •

  P1: This annotation policy is usually used in semantic landmark localization tasks, especially in facial landmark localization. Specifically, the rectangular area of the bounding box tightly encloses a set of pre-defined facial landmarks.

• •

  P2: This annotation policy has been widely used in face detection datasets [74]. The bounding box contains the areas of the forehead, chin, and cheek. For the occluded face, the bounding box is estimated by the human annotator based on the scale of occlusion.

Qualitative Analysis. As shown in Fig. 8, we present some “good” and “bad” facial landmark detection examples according to NME. Specifically, for the good cases presented in the first row, most images are of high quality; For the bad cases in the second row, most images contain heavy blurring and/or occlusion, making it difficult to accurately identify the contours of different facial parts.

We perform experiments on two popular human pose estimation datasets,

• •

  MPII [85]: The MPII Human Pose dataset contains around 28,821 images with 40,522 person instances, in which 11,701 images for testing and the remaining 17,120 images for training. Following the experimental setup in  [22], we use 22,246 person instances for training and evaluate the performance on the MPII validation set with 2,958 person instances, which is a heldout subset of MPII training set.
  

• •

  COCO [86]: The COCO dataset contains over 200,000 images and 250,000 person instances, in which each person instance is labeled with 17 keypoints. Following the experimental setup in [22], we evaluate the proposed method on the validation set with 5,000 images.

We utilize recent state-of-the-art heatmap regression method for human pose estimation, HRNet [22], as our baseline. Specifically, the proposed quantization system can be easily integrated into most heatmap regression models and we have made the source code of human pose estimation based on the HRNet baseline publicly available. For the MPII Human Pose dataset, we use the standard evaluation metric, head-normalized probability of correct keypoint or PCKh [85]. Specifically, a correct keypoint should fall within $\alpha*l$ pixels of the ground truth position, where $l$ indicates the normalization distance and $\alpha\in[0,1]$ is the matching threshold. For fair comparison, we apply two different matching thresholds, [email protected] and [email protected], where a smaller matching threshold, $\alpha=0.1$, indicates a more strict evaluation metric for accurate semantic landmark localization [22]. For the COCO dataset, we use the standard evaluation metric, averaged precision (AP) and averaged recall (AR), where the object keypoint similarity or OKS is used as the similarity measure between the ground truth objects and the predicted objects [86].

The experimental results on the MPII dataset are shown in Table IX. Specifically, when using a coarse evaluation metric, [email protected], both the proposed method and the compensation method achieve comparable performance to the baseline method, suggesting that the quantization error is trivial in coarse semantic landmark localization; When using a more strict evaluation metric, [email protected], the compensation method, which can be seen as a special case of H3R with $k=2$, significantly improves the baseline, e.g., from $32.8$ to $37.7$, while the proposed method H3R further improves the performance from $37.7$ to $39.3$. The experimental results on the COCO dataset are shown in Table X. Specifically, the proposed method clearly improves the averaged precision (AP) in different settings and the major improvements on AP come from: 1) a strict evaluation metric, e.g., AP^0.75; and 2) large/medium person instances, i.e., AP(M) and AP(L). Furthermore, we also find that the improvement decreases when increasing the input resolution, e.g., from $0.674$ to $0.720$ for $192\times 128~{}(0.046{\uparrow})$, from $0.723$ to $0.750$ for $256\times 192~{}(0.027{\uparrow})$, and from $0.748$ to $0.762$ for $384\times 288~{}(0.014{\uparrow})$.