CLERF: Contrastive LEaRning for Full Range Head Pose Estimation

Given a training sample, we generate with 3D-aware GAN a synthetic image that has the same yaw and pitch values and apply proper rotation to match the roll value (Figure 1 step 1). With the mathematics listed in Hu et al. [15], we can solve for a triad of $(yaw,pitch,\phi)$ from the rotation matrix. The rotation associated with the roll angle $\phi$ based on the rotation matrix $R\in SO(3)$ (refer to [33]) representing rotation with yaw and pitch is the following:

where $R_{rotate}$ is the rotation matrix of the rotated image. We provide detailed formulation below.

a novel two-stage self-adaptive image alignment for robust training, a tri-grid neural volume that effectively manages the representation challenges of both the face and the back of the head, and the integration of prior knowledge from 2D image segmentation to improve 3D model training. Panohead allows the generation of a head pose facing a specific yaw and pitch, which is a crucial component in facilitating contrastive learning. As a starting dataset, we generated 25,984 images with 812 unique-looking people and 32 images each. Each set of 32 images is uniformly distributed within the yaw and pitch space of [-3.14 radians, 3 radians] and [-1.5 radians, 0.1 radians]. The rotation matrix $R$ follows the reference system of 300W-LP as below:

The second key lies in the geometric transformation on head pose images and their corresponding rotation matrices to enable FRHPE (Figure 1 step 2). Besides rotation defined in Equation 1, we follow the mathematics in Hu et al. [15] and apply flipping across $L_{\theta}$ on the XY-plane with a given rotation $R\in SO(3)$ to obtain the following rotation matrix:

where $\theta$ is the counter-clockwise angle between the positive x-axis and the line $L$ that we flip across. Figure 2 shows the resulting distribution of the 300W-LP dataset after our rotation and flipping (in red), compared to that of random rotation matrices (in green) and the original distribution (in blue). As the 300W-LP dataset lacks head poses facing away from the camera (large yaw angles), the resulting distribution does not fully match the uniform distribution (a semi-sphere versus a sphere in Figure 2(c)). Nonetheless, our rotation and flipping augmentation fills the gap and transforms head poses to cover a much wider range than the original images.

To maintain the guaranteed existence of a positive pair, we apply the same rotation and flipping augmentation to both images in a positive pair. Since this is a crucial condition for contrastive learning, we prove Theorem 3.1 in Appendix A that geometric transformations preserve the geodesic distance $d$ for any two rotation matrices $A,B\in SO(3)$, with $d$ defined as the following [12]:

where $tr$ and $T$ represent the trace and transpose, respectively. In other words, for any chosen anchor, positive, negative triplet $(a,p,n)$, applying the same rotation and flipping augmentations results in another valid triplet $(a^{\prime},p^{\prime},n^{\prime})$. We present our strategy for sampling triplets for contrastive learning.

After data augmentation, we pass the batch of images to the image encoder $E$ to obtain their embedding vectors (Figure 1 step 3) and form triplets. Though the image generation step guarantees an anchor-positive, we include neighboring head orientations that have a geodesic similarity $cos(d(R,R^{\prime}))$ higher than the threshold $T_{GD}$ as additional anchor-positives, where $R$ and $R^{\prime}$ are the rotation matrices of the anchor and another image in the mini-batch (Figure 1 step 4). Furthermore, we filter and retain the hard and semi-hard negatives that violate a margin $v$ according to the Euclidean distance between the embedding vectors. Subsequently for back-propagation (Figure 1 step 6), we compute the Circle Loss [30] $L$ (Figure 1 step 5) that aims to increase the similarity between positive pairs and decrease that between negative pairs, through a re-weighting process to focus on the less-optimized samples, with $L$ defined as the following:

where $\gamma$ is a scaling factor, $m$ is a relaxation margin, $s$ is the cosine similarity of the embedding vectors, $N_{p}$ and $N_{n}$ are the number of positive and negative pairs respectively.

After the representation model is trained, we freeze it and train a downstream multi-layer perceptron (MLP) to project the representation to the fine-grained head pose angles, represented by a rotation matrix (Figure 1 step 7). To ensure a unitary matrix, we transform the six-dimensional MLP output to the nine-dimensional rotation matrix through the Gram-Schmidt process as implemented in 6DRepNet [12]. With a trained representation model and its downstream MLP, we evaluate test datasets and compare against baseline models.

Training. 300W-LP [43] contains 122,450 images from multiple databases with faces mainly in the frontal range. We utilize PanoHead [1] as the 3D-aware GAN to generate anchor-positives for the representation model training only.

Testing. Following previous works, we test the models on AFLW2000 [43] and BIWI [9]. These datasets mainly contain frontal faces with yaw in the range of [-90, 90] degrees. Therefore, to facilitate a comprehensive analysis of our full range capability, we additionally evaluated on four variants of the previous test sets: slightly-augmented (SA) AFLW2000, SA BIWI, fully-augmented (FA) AFLW2000, and FA BIWI. The SA version is obtained by rotating each image clockwise by 10 degrees and flipping it along the line that is 85 degrees counter-clockwise from the positive x-axis (an example is shown in the second row of Figure 4). As for the FA version, we randomly rotate between -180 to 180 degrees and flip along the line that is between 0 to 90 degrees counter-clockwise from the positive x-axis (an example is shown in the third row of Figure 4).

Representation Model. We use the improved version of Swin Transformer Base model [22] with weights pre-trained on ImageNet in PyTorch and an output embedding of size 1024 as the image encoder $E$. We train it for 30 epochs on 20,000 images from 300W-LP dataset and 20,000 PanoHead-generated images with the Adam optimizer and a learning rate of $10^{-5}$ on a single NVIDIA Tesla V100 32GB GPU. We utilize the PyTorch Metric Learning library [25] for triplet sampling and loss computation. For hyperparamter values, we apply $T_{GD}=0.8$ and $v=0.1$ for triplet filtering and $m=0.4$ and $\gamma=80$ for loss $L$.

MLP. Our downstream MLP consists of four fully connected layers of 256 units and a skip connection from the input layer to the last fully connected layer. We train the MLP for 40 epochs with mean geodesic distance $d$ as the loss function and an exclusive subset of the training dataset as the validation set for early stopping.

Data Augmentation. With a probability of 0.5, we randomly rotate the image clockwise by a degree between 0 and 90, and with another probability of 0.3, we randomly flip the image along an axis that is between 0 and 90 degrees counter-clockwise from the positive x axis. For pixel-wise augmentation, we deploy the popular techniques including translation, resizing, down-sampling for low resolution, hue change, sharpness, grayness, contrast limited adaptive histogram equalization, and brightness. Furthermore, we conduct the following pixel-wise transformations: RGB shift, channel shuffle, gamma correction, color jitter, Gaussian noise, and Gaussian blur. For pixel removal, we utilize center crop and coarse dropout. These methods alter the pixel values, reinforcing a more robust training outcome.

Baseline Models and Evaluation Metric. We consider many existing HPE models as our baseline models. FSA-Net [36] utilized feature aggregation and soft stagewise regression. HopeNet [26] applied multiple losses on a convolutional neural network. TokenHPE [37] leveraged a transformer to learn the relationship within the facial part. 6DRepNet [12] relied on the geodesic loss function and its variant, 6DRepNet360 [12], expanded to full range. WHENet [40], the other full range model, wrapped the loss function to stabilize the learning of large yaw angles. Following these works, we evaluate the models with mean absolute error (MAE) on each of yaw, pitch, and roll and compute the average of the three MAEs as the Mean. Since no models, including ours, are specialized in one of yaw, pitch, or roll predictions, we mainly compare the Mean.

We present the evaluation of all models on six datasets in Table 1. On AFLW2000, CLERF performed nearly on par with 6DRepNet, down by 1.4 degrees in Mean. On BIWI, CLERF performed on par with TokenHPE, falling short by 0.01 degree in Mean. We recognize that having FR coverage, CLERF optimizes for the entire range, leaving relatively less focus on the frontal range that non-full range models directed all of their resources to learn. We act on this to construct a more comprehensive analysis by slightly rotating and flipping the same test images (SA AFLW2000 and SA BIWI), keeping them within the same frontal range that non-full range models trained on. Results show that CLERF outperformed all baseline models, a sign that existing models might have been fixated on the exact angles of the original test images. With 0.57 and 1.29 degrees better than the runner-up for SA AFLW2000 and SA BIWI, respectively, CLERF demonstrates robustness and superiority. Additionally, we examine the full range capability by rotating and flipping the test images on a larger scale and only compare CLERF to full range baseline models. CLERF remains the highest performing, leading by more than 10 degrees in Mean. An interestingly observation is that CLERF had a minor improvement of 0.03 degrees in Mean on FA AFLW2000 compared to the original version, an evidence that CLERF optimizes for the entire range.

Contrastive Learning. We validate the contrastive learning motivation by comparing against the traditional supervised learning approach. We concatenate the representation model with the downstream MLP and train on geodesic loss with the same dataset. The comparison result is shown in Table 2, and we observe a significant improvement with our contrastive learning approach. This demonstrates that contrasting between neighboring angles and further angles boosts the learning of head orientations. By first grouping up the neighboring head poses and separating those that are further away, we enable the downstream MLP to more accurately project the representation to fine-grained head poses.

Batch Size. Batch size is a crucial hyperparameter in contrastive learning when anchor-positives and anchor-negatives are identified within a mini-batch. With sparsity in head orientations, a large batch size tends to increase the number of anchor-positives and the ratio of anchor-positives to anchor-negatives. Figure 3(a) shows the model performance across different batch sizes. We observe a consistent pattern across all four datasets: a larger batch size results in better performance up until a batch size of 32, which corresponds to 32 images from 300W-LP and 32 PanoHead images generated on-the-fly.

Epochs. As the representation model aims to learn a good representation, we study how the number of training epochs affects the learned representation, with results shown in Figure 3(b). Model performance peaked at 16 epochs and declined afterwards. This is likely due to the limited varieties appearing in the synthetic images generated through PanoHead that accumulated over the number of epochs.

Image Encoder Backbone. While we select the Swin Transformer as our image encoder for many of its strengths, we compare to two other backbone models to analyze our proposed approach across different image encoders, as shown in Figure 3(c). The Swin Tranformer consistently outperformed others, with ResNet50 [11] generally performing better than the EfficientNetB0 model [31], which aligns with their descending number of model parameters.

Rotation and Flip Augmentation. An important component in our approach that enables FRHPE is the rotation and flip as data augmentation techniques. To quantify their significance, we present the respective improvements in Table 3. The combination of rotation and flip performed the best, with rotation or flip alone making their respective contributions. These geometric transformations resulting in better performance not only in AFLW2000 and BIWI, but also in their SA versions.

Qualitative Analysis. After the empirical results, we visually illustrate our model performance with three test cases, each representing one from the original, SA, and FA versions of AFLW2000 test set. We compare to the ground truth and baseline models, as shown in Figure 4. The first row presents an original image with very similar head pose predictions across all models, which is consistent with the similar Mean values listed in Table 1. Second row presents the SA version of the same image and we observe more noticeable deviation for WHENet and TokenHPE on the blue line. As for the FA version of the original image shown in the third row, CLERF fully adapted to the heavy rotation while the baseline models struggled, despite that 6dRepNet360 and WHENet aimed at being full range models. This illustrates that although CLERF is on average one degree behind the baseline models on the original AFLW2000 test set, CLERF adapts and produces accurate results in the full range, making the one degree negligible, not to mention the other test set, BIWI, where CLERF performed on par with baseline models on the original images and outperformed them in the SA and FA versions.

TSNE Visualization. For comprehensive analysis, we take a further step to visualize with 3D t-distributed stochastic neighbor embedding (TSNE) the embedding vectors of a sequence of images showing a person turning in a full circle. Figure 5 compared to the supervised and other baseline models, using a cyclic color map that visualizes neighboring frames with similar colors. An optimal pattern consists of a clear circle with the colors following the cyclic order. CLERF (Figure 5(e)) exhibited clear separation for angles that are further away and kept nearby angles close as shown by the continuous change of colors. On the other hand, lacking only the contrastive learning component, the supervised model (Figure 5(d)) showed the continuity in color changes but did not separate opposite angles as effectively. We made a similar observation with the other baseline models (in Figure 5(a)$\sim$(c)).