Semi-supervised Contrastive Regression for Estimation of Eye Gaze

Eye gaze is one of the most widely used human mode for developing human-machine interface system. Gaze controlled interface has become quite popular for virtual reality (VR) applications [1], navigation control of ground and aerial robots [2], control of robotic arms for surgery [3] and other commercial applications. With the advances in deep learning methods, gaze tracking has become an achievable task.

Appearance-based models are the most prominent approach that does not require an expensive and subject-dependent eye modeling. It mainly relies on annotated face images captured using camera/eye-tracker and learns a model to map gaze direction from the images. Most of the earliest methods relied on dedicated feature extraction [4][5] and feature selection steps, before performing the regression task [6]. This in effect makes the models more person-specific and generalization is not attained. At the same time, dedicated feature extraction also makes the computation time-taxing. With the advancements in deep learning [7], convolution neural networks(CNN) based frameworks [8][9] have become the most used architectures for gaze estimation. Mostly ResNet based CNN frameworks [10] have been developed that maps the gaze angles on the eye images. Dual branch CNN [11] has been another adopted approach for gaze estimation that reconstructs images with the gaze direction in supervision. Sequential models have also been used to learn the variation in gaze direction in subsequent frames, by leveraging LSTM models [12] on the residual features obtained using CNN. Capsule networks are another most promising approach for appearance-based models [13][14] as it emphasizes on learning equivariance instead of finding deeper features.

Generalization of appearance-based models is solely dependent on the volume of labeled data. Annotation of gaze direction in eye images is a difficult and time-consuming task. Even with the large gaze datasets available in the public domain, there are still issues in developing a domain-adaptive model due to varied background environment and illumination conditions. To deal with these problems, Semi-supervised learning(SSL) is one of the most promising solution.

SSL performs a pre-training [15] on the unlabeled data to learn an effective representation of the input. It further fine-tunes the solution using a small labeled dataset. Contrastive learning, a SSL technique, has been predominantly used in different computer vision applications in past few years. Contrastive learning based methods learn semantic embedding representation [16] of the input image by pulling the similar images together and pushing the disparate images away [17]. This facilitates the models to learn a suitable encoding of the images [18] using only the unlabeled data. Consequently, a limited gaze annotations are used to learn a model that can perform the final task of classification, segmentation or prediction. Wang [19] have first introduced the use of contrastive learning method for unsupervised regression learning of eye gaze direction.

In this paper, we present a semi-supervised regression technique to predict the eye gaze direction with two major contributions.

• •

  We have developed a contrastive learning framework that learns an encoder architecture to compute embeddings and further uses the pre-trained embeddings to predict the gaze directions.

• •

  We have proposed a new form of contrastive loss to maximize the agreement between similar images that can take care of both invariance and redundancy factors simultaneously.

The designed framework is based on SimCLR framework [16] that performs the task in two stages. The first stage aims to learn an representation from the given images by maximizing the agreement between the vector representations learned from the images. The final stage uses the embeddings learnt in the pre-training stage and trains a model that performs the task of prediction by minimizing a loss function.

Mini-batches of batch size $B$ are sampled from the dataset $(X,G_{dir})$, where $X$ represents the images and $G_{dir}$ represents the gaze direction labels for those images. Random data augmentation is performed to create two varied representations of the images in the mini-batches $X_{a1}$ and $X_{a2}$. We have designed an encoder module $\mathcal{E}$ that learns latent space representations $f_{1}=\mathcal{E}(X_{a1})$ and $f_{2}=\mathcal{E}(X_{a2})$ from the augmented images. Latent space embeddings learns high-level features from the images. These embeddings are then passed to a projection head $\mathcal{P}$, which is designed as a multi-layer perceptron (MLP) to learn a non-linear projection vector for less complex processing. The learned embeddings are $p_{1}=\mathcal{P}(f_{1})$ and $p_{2}=\mathcal{P}(f_{2})$, which are then compared to minimize the contrastive loss for similar image pairs.

The designed encoder can learn local as well as global dependency in feature maps. The architecture of the encoder is shown in Fig.1(a). In order to learn global spatial dependency along with local spatial details, larger kernel size needs to be considered. But, larger kernel size increases the computation complexity of the model and can also lead to overfitting. Thus we have used dilated convolution with different dilation rates and local spatial dependency is learnt with each convolution. By using dilated convolution, we were able to attain larger receptive fields without increasing the computation complexity due to larger kernel size by using different dilation rates. Feature map of high spatial resolution is obtained by concatenating the coarse to fine feature maps learned. Convolution filters for different dilation rates have been shown in Fig.1(b), where each colour indicates kernel corresponding to each dilation rate in concatenated feature map.

An updated version of the Normalized temperature scaled cross-entropy (NT-Xent) loss [16] has been used to define our contrastive loss. The aim is to minimize the invariance in data as well as the redundancy in feature maps at the same time. The NT-Xent loss aims to maximize the similarity between two positive images and minimize the similarity between a positive and a negative image. We have used the concept of computing cross-correlation matrix as introduced in Barlow Twins [18] to analyze the correlation between the positive and negative images. The cross-correlation between the projections $p_{1}$ and $p_{2}$ is computed as $C$, where each element basically represents the similarity between each element of the projections. The contrastive loss defined in this paper is displayed in Equation [1], where the first term defines the NT-Xent loss that tries to minimize the invariance, whereas the second term reduces the redundant information in the output vector representation. The term $\gamma$ represents loss coefficient factor to consider the redundancy term which has been set to $\gamma=0.01$. The terms $sim(p_{i},p_{j})$ defines the cosine similarity between $p_{i}$ and $p_{j}$ and is defined by Equation [2].

The framework pre-training stage and fine-tuning stage for regression is shown in Fig.2(a) and Fig.2(b) respectively. It uses the pre-trained encoder model and determines the values of latent space representations $f_{i}$. The latent vectors are used as an input to the regression network to predict the values of gaze directions $G_{pred}$. The regression fine-tuning network is optimized by minimizing the huber loss, which is a step-wise amalgamation of mean squared error and mean absolute error to tackle the outlier issue, as defined in Equation [3] where $d=G_{pred}-G_{dir}$. The loss parameter $\delta$, that provides a measure of spread of the deviation error, ensures the loss function to generate large value of loss, by computing the squared value of deviation, for larger deviation values and smaller values of loss, by computing linear mean absolute error, for small deviation values.

This work has enlightened on semi-supervised learning for eye gaze prediction tasks by developing contrastive learning framework. The work has proposed a new form of contrastive loss to optimize the similarity agreement between augmented and captured face images considering two parameters, viz., invariant transformation and redundancy among images, to predict gaze direction. The proposed model has been tested on ETH-XGaze dataset. The evaluation of the proposed model has been done in terms of mean angular error. The model has outperformed different state of the art techniques such as SimCLR, Barlow twins as shown in the previous sections. In future we will experiment across different datasets and find a more generalized solution for cross-datasets scenarios.