CLRGaze: Contrastive Learning of Representations
for Eye Movement Signals

Our methodology is largely based on SimCLR [12]. For any image $x\in\mathbbm{R}^{d_{x}}$ in a mini-batch of size $N$, two different random image augmentations are performed (e.g. cropping, blur, color distortion) to obtain two new sub-images $x_{a}$ and $x_{b}$ that form a positive pair. This doubles the mini-batch size to $2N$, where each image now has one positive example and $2(N-1)$ negative examples.

A CNN $f$ encodes this mini-batch $\{x_{k}\}\in\mathbbm{R}^{d_{x}}$ to their representations $\{h_{k}\}$ in a learned latent space $\mathbbm{R}^{d_{h}}$. The representations are further mapped by a nonlinear projection head $g$ to a small feature vector $\{z_{k}\}\in\mathbbm{R}^{d_{z}}$ with which their intra-batch similarities are calculated. The encoder $f$ and projection head $g$ are jointly optimized such that the similarity between positive pairs are maximized while that of negative pairs are minimized. Specifically, they minimize the normalized temperature-scaled cross entropy loss (NT-Xent). For any data point $i$ and $j$ in the mini-batch, NT-Xent is computed as follows:

where $sim$ is the cosine similarity between two projections $sim(z_{i},z_{j})=z_{i}^{T}z_{j}/\|z_{i}\|\|z_{j}\|$, $\mathbbm{1}$ is an indicator function evaluating to $1$ when the data point is not being compared to itself, and $\tau$ is a temperature parameter that scales the similarity values. By training on larger batches and many iterations, the network is exposed to more positive and negative examples. Thus, allowing it to form richer abstractions about the data.

We take inspiration from SimCLR and apply this framework to eye movement signals. We port their methodology to the signal or 1D time-series domain by performing a set of data transformations analogous to augmentation techniques for images or 2-D data.

A sample signal $x\in\mathbbm{R}^{(2,T)}$ is a time-series with arbitrary length $T$ and 2 channels corresponding to the x and y plane Each time step is an estimated position of a viewer’s gaze on a screen. To produce a positive pair $(x_{a},x_{b})$ from $x$, we must obtain two different views or segments $x_{a},x_{b}\in\mathbbm{R}^{(2,T^{\prime})}$ where $T^{\prime}\leq T$ is a fixed input length. We do this by randomly selecting any of the three cropping methods visualized in Figure 2.

Given $T^{\prime}$ and a randomly selected time point $t$ where $t\leq T^{\prime}\leq T$, the methods are as follows: In (1) Same, the two segments are identical ($x_{a}=x_{b}=x_{t:t+T^{\prime}}$). In (2) Consecutive, the two segments are consecutive portions of the signal, i.e. $x_{a}$ immediately proceeds or precedes $x_{b}$ (e.g. $x_{a}=x_{T^{\prime}-t:t}$ and $x_{b}=x_{t:t+T^{\prime}}$). In (3) Random, the two segments come from any random portion of the signal (e.g. $x_{a}=x_{t_{1}:t_{1}+T^{\prime}}$ and $x_{b}=x_{t_{2}:t_{2}+T^{\prime}}$). After cropping, we separately apply, with uniform probability, one out of nine transformations listed in Table I to both segments. The transformations alter or destroy the signal encouraging the network to find unique patterns and be robust to noise.

Our encoder $f$ is a six-layer temporal convolutional network (TCN) [14] with residual and squeeze-and-excitation blocks [15, 16] detailed and visualized in Figure 3. Following SimCLR, our projection head is a multilayer perceptron (MLP) with one ReLU-activated hidden layer.

We train two networks: CG, which is trained on all available data and CG-3, a model trained only on EMVIC, FIFA, and ETRA data sets. We do this to have a fair comparison with the velocity models of GazeMAE (GM_v) [11], a previous work on eye movement representation learning using deep convolutional autoencoders. CG and CG-3 employ all cropping methods and data transformations, and have the same network architecture described in Section II-B and Figure 3.

Our inputs are 1-second segments or 500 time points ($T^{\prime}=500$). Our encoder $f$ then maps a sample signal $x\in\mathbbm{R}^{(2,500)}$ to its representation $h\in\mathbbm{R}^{512}$. We train our networks to minimize the NT-Xent Loss (Eq. 1) with $\tau=0.3$, learning rate=5e-4, batch size=1000 and Adam optimizer [22]. CG-3 is trained for 100 epochs while CG is trained for 800. For a training set of 45,755 samples, this results to 45 batches per epoch. We train for 800 epochs or 36,000 iterations. We notice that performance does not improve beyond this. The networks are implemented with PyTorch 1.7 [23] and trained on an NVIDIA RTX 2080Ti GPU. We compute with automatic mixed precision (AMP) to enable larger batch sizes and faster training time. All parameters are chosen empirically, and a random seed was set for all experiments. Our code will be made available at https://github.com/chipbautista/clrgaze.

We use the trained network to encode eye movements into feature vectors. We now input full-length samples $x\in\mathbbm{R}^{(2,T)}$ instead of fixed-length segments used for training. The GAP layer allows our encoder to handle arbitrary lengths, making it a more practical approach. The feature vectors are then used as inputs to a linear classifier, which is a standard evaluation method for representation learning [24]. In our case, our classifier is a linear Support Vector Machine (SVM) implemented through scikit-learn library [25].

Limited by the available data labels, we opt to evaluate the learned representations by classifying the viewers based on their eye movements. This is also known as eye movement biometrics, a growing research area that, if done with traditional feature extraction methods, requires extensive knowledge on eye movements [6]. When possible, we compare our results with other works that have conducted the same tasks.

From Table III, it is shown that CLRGaze outperforms the previous works in all tasks. We believe that this boost can be attributed to our methodology. Recall that the contrastive CNN has to correctly classify if two segments originated from the same eye movement signal. To do so, it has to extract the patterns that are present throughout the signal, which are patterns that are likely to be idiosyncratic or unique to the viewer. Applying random cropping and chunk transformations to the eye movements further encouraged the CNN to extract these global information patterns. This concept is related to slow feature analysis and contrastive predictive coding [26, 27].

Also, notice that substantially lower accuracies were achieved for the MIT data sets, which may indicate that 240Hz eye-trackers cannot capture granular information needed for biometric tasks.

Next, we train more models with the same parameters, changing only the composition of data transformations. We evaluate on the Biometrics (All) task since we deem this the most difficult. From Table IV, it is shown that the choice of cropping methods and data transformations largely impacts accuracy. While we present only the results for Biometrics (All) for simplicity, note that we observed the same trend when evaluated on other tasks.

Finally, we train a model with the same parameters but on a viewer-stratified split (22,877 training and 22878 validation samples). In Figure 4, we plot the representations of the validation set, using the viewers as the labels. Eye movements of a subject lie close together in the representation space, suggesting that our model can represent unseen samples sufficiently.