Benefits of temporal information for appearance-based gaze estimation

In this work, the spatio-temporal gaze estimation task is posed as a regression problem based on monocular eye images. First, spatial features are extracted for each frame $I_{i}$ of a sequence using a static CNN backbone $g$. The sequence of per-frame features is then fed to a many-to-one recurrent module $r$ to learn sequential dependencies. The recurrent module produces a vector of spatio-temporal features, which is used to finally regress the line of gaze of the last frame of the sequence, $\hat{y}_{t}$, such that $\hat{y}_{t}=f(r(g(I_{t-s-1}),\dots,g(I_{t})))$, where $f$ denotes the regression function, $t$ corresponds to the last frame of the sequence, and $s$ to the number of previous frames considered for the sequence. The line of gaze is expressed by 2D spherical coordinates, representing yaw (horizontal) and pitch (vertical) angles.

As backbone, we use a modified ResNet architecture  [4] with 13 convolutional layers and a final adaptive average pooling layer at the end to decrease the final feature vector size to 64x4x4. The sequence of feature vectors is flattened to serve as input for the recurrent module. The backbone architecture is depicted in Fig.  1.

The recurrent module consists of a single-layer plain Long Short-Term Memory (LSTM)  [1] with 32 units. The internal memory cell of an LSTM is able to learn long term dependencies of sequential data, avoiding vanishing and exploding gradient problems. The LSTM is unrolled into $s$ time steps, depending on the input sequence length. The output of the recurrent module is further fed to a fully connected (FC) layer with Rectified Linear Unit (ReLU) activation function, which produces a 32-dimensional vector. Finally, a FC layer with linear activation function (i.e. regression) produces the estimated 2D gaze angles.

The network is trained in a stage-wise fashion. First, the static backbone, coupled to a 32-hidden unit FC layer and a 2-hidden unit FC regression layer, is trained end-to-end from scratch on each individual frame of the training data to learn static features. This network is referred to as Static1 (or S1) in Section III. Second, the FC layers are discarded, and the recurrent module, coupled to a new 32-hidden unit and 2-hidden unit FC layers, is added to the pre-trained static backbone. The new architecture is further trained end-to-end, fine-tuning the convolutional backbone while training recurrent layer and new FC layers from scratch. By further fine-tuning the backbone weights, the convolutional module is able to learn useful features coming from the sequential information captured by the recurrent module. For this second stage, however, the training data is re-arranged using a sliding window with stride 1, to build input eye image sequences compatible with the many-to-one architecture. Each input sequence is composed of $s$ consecutive frames. This second network is referred to as S1+LSTM in Section III.

The network was trained using ADAM optimizer [7], empirically setting the learning rate to 0.0005, batch size to 32, and weight decay to 0.00001. The learning rate parameter was found to have a large influence on the final accuracy, with lower values not allowing a proper learning of the LSTM. CNN weights were initialized from a uniform distribution. For the LSTM module, input weights were initialized using Xavier uniform, while an orthogonal initialization was used for hidden weights. Biases were set to 0. Early stopping on the validation set was used to select the best model for each training stage, with a maximum number of epochs of 150 for the first stage and 30 for the second. Finally, we used the L1 loss for both training stages, as preliminary experiments showed it to yield slightly lower error than L2 loss.

The study is based on a newly constructed dataset of $100\times 160$-pixel eye-image sequences captured using a VR-HMD ¹¹1Details on capture setup are beyond the scope of the short paper and will be discussed in a follow-up long form version of the paper. The VR-HMD used was similar to usual commercially available wireless VR-HMDs., with two synchronized eye-facing infrared cameras at a frame rate of 100Hz under constant illumination. The dataset consists of 84 subjects with appearance variability in terms of ethnicity, gender, and age, with some of them wearing glasses, contact lenses, and/or make-up. Subjects were recorded while gazing at a moving target on screen. Each recording consisted of a set of patterns with 1s-long randomly-located target fixations at different depths and instantaneous (0.1s) target transitions to elicit saccades.

Ground truth eye gaze vectors were obtained using a classical user-calibrated glint-based model [2]. While this approach poses some limitations on the ground truth quality (see Section IV), it still allows us to soundly evaluate our hyphoteses. Frames with no valid gaze vector, or for which the subject was distracted, were discarded, causing most of the recordings to be divided in smaller non-contiguous frame sequences. To keep consistency, the remaining data was further processed by randomly selecting 10 non-overlapping sequences of 100 contiguous frames (1s) each, thus having 1,000 frames per recording and a total of 168,000 eye-region images. Therefore, each sequence can contain fixations only, saccades, or a combination thereof. Fig. 2 shows the gaze angle distribution and sample eye images from the dataset.

To perform the experimental evaluation, the processed dataset was partitioned into subject-independent train, validation, and test sets following a 5:1:2.4 ratio. The evaluated models were trained on the training split, using the validation split to optimize model hyper-parameters. Right-eye images and their corresponding ground truth gaze vectors were horizontally flipped to mimic left-eye data. This way, the same model can be used for both eyes, while augmenting the number of data samples.

Experimental results are reported on the test split, using Mean Absolute Error (MAE) between estimated and ground truth 2D gaze angles as main metric. Due to the sliding window approach followed by the spatio-temporal model, the first frames of a sequence do not obtain a gaze estimation. Therefore, in this section, results are reported on the subset of the test split which does obtain gaze estimates for all evaluated models ²²2The longest window length evaluated in this paper is 20 frames; therefore, $81\%$ of the test split is used to report experiment results. Results from models based on smaller window lengths, evaluated on a consequently larger test subset, did not significantly deviate from the results reported herein..

First, we use the initial static model (Static1) as baseline and compare it to the proposed spatio-temporal model. In particular, we evaluate 4 versions of the spatio-temporal model, each of them trained with different sliding window lengths $s$ in the range $\{5,10,15,20\}$, to assess the effect of the amount of frames used on the final accuracy.

Table  I shows the performance of the evaluated models with respect to each axis individually and simultaneously. We can observe that all spatio-temporal models significantly outperform the static baseline, with up to $19.78\%$ mean error improvement (paired Wilcoxon test, $p<.0001$). While the error for the horizontal axis is higher than the vertical for all models, the addition of temporal information decreases the error by up to $16.91\%$ for the former and $23\%$ for the latter, evidencing that such information is more beneficial for the vertical axis. This is an important contribution with respect to classical or pupil-based methods, as they usually have less accuracy on this axis due to occlusions of the eye limbus caused by eyelids and lashes. With respect to the window length, we can observe that the increase in performance peaks around $s=15$ frames (i.e. 150ms) and then diminishes, indicating that longer-term dependencies are not required to obtain further accuracy gains.

It could be argued that the decrease in error is due to the larger complexity of the spatio-temporal models, as the addition of the LSTM layer highly increases the number of parameters. To validate this possibility, we trained a second static model (Static2 in Table  I), adding a 128-hidden unit FC layer between the two FC layers from Static1 model to compensate for the difference in number of parameters between baseline and spatio-temporal models. Results show that, in spite of the smaller number of parameters, even Static1 outperforms Static2, suggesting that Static2 may be overfitting to the training data. This demonstrates that the increase in complexity is not correlated with a lower error.

Fig. 3 further illustrates the effects of leveraging temporal information, with an example of ground truth and estimated gaze angles during a fixation-saccade transition, for horizontal and vertical axis. We can clearly see the smoothing effect of temporal information as opposed to the noisy static estimation. Furthermore, spatio-temporal estimates are able to more accurately follow the saccade-to-fixation transition. Indeed, using consecutive frames allows the network to better discard noisy features and learn a more useful representation for eye gaze estimation.

In this subsection, we evaluate the contribution of temporal over static models with respect to different eye movements types. To do so, we use the 20-frame-window spatio-temporal model (S1+LSTM20) as the reference temporal model for this experiment.

Since our dataset has a mixture of eye movements, to perform this evaluation we manually annotated the test split based on visual inspection of ground truth gaze angles with the following labels per each 20-frame input sequence: Fixation, when the eye was virtually static; Saccade, if it only included a saccade movement; Transition, if it included a combination of fixation and saccades; and Other, when the eye status could not be clearly classified. Transition sequences were further divided into Fixation to saccade, Saccade to fixation, or Fixation to saccade to fixation, according to their order in time.

Fig. 4 depicts the contribution of temporal information with respect to the static baseline for each label and axis. As shown previously, the performance of the vertical axis substantially improves when adding temporal information for fixations and saccades, compared to the horizontal axis. In particular, we observe a substantially higher improvement for saccades, with an average difference of 0.44 degrees between axes. With respect to transitions, the horizontal axis shows a higher improvement than the vertical axis for fixation-to-saccade transitions. This demonstrates that the spatio-temporal model is indeed taking into account the information coming from the first frames of the input sequence, being more beneficial when the sequence starts with a more stable eye gaze and more correlated eye images to better learn the transition dynamics. As a matter of fact, this improvement is even higher than the one obtained on Saccade-only sequences, suggesting that the model is able to learn more representative features of the eye when being presented with a fixation bout first, particularly for the horizontal axis.

As a final experiment, we evaluated the differences on reported error for each subject. While the error indeed varied among subjects, we did not find any significant differences with respect to the subjects characteristics (i.e. age, gender, glasses and make-up).