Relax DARTS: Relaxing the Constraints of Differentiable Architecture Search for Eye Movement Recognition

Traditional methods typically use manual techniques to filter useful features from eye movement data and then apply ML algorithms for feature extraction and recognition. Common features include eye movement velocity, gaze duration, and path length.

In 2004, Kasprowski[15] proved that eye movement data contains information that can be used for identity recognition firstly. In 2008, Komogortsev[16] fully considered the bioanatomic properties of the eyeball and established a linear horizontal oculomotor plant mechanical model, [17] expanded on this foundation and proposed the concept of oculomotor plant characteristics (OPC). In 2017, Bayat[18] conducted recognition experiments on a self-constructed dataset, using eye movement data combined with pupil size. In 2018, Li[19] and colleagues extracted eye movement features using a multi-channel Gabor wavelet transform.

DL models are extensively utilized in computer vision (CV)[20] and natural language processing (NLP)[21], but also in the field of eye movement recognition, due to their capacity for end-to-end feature learning.

In 2019, Lena[22]developed a convolutional network (CNN)-based Siamese Network that utilizes eye movement data as input into two separate sub-networks for recognition. These studies[23] fine-tuned the model and expanded it into the DeepEyedentificationLive (DEL) model. In 2022, Dillon et al.[24] proposed exponentially-dilated CNNs for recognizing eye movement features. In addition, in 2023, Taha[25] et al. collected vehicle driver eye movement data via a remote low-frequency acquisition device, and extracted data features by combining Long Short Term Memory (LSTM) and dense networks to achieve end-to-end driver identification. In the same year, Qin et al.[5] extracted temporal and spatial features of eye-movement data for the combination using improved LSTM and Transformer algorithms to achieve state-of-the-art performance.

DARTS[11] uses $Softmax$ to treat the selection of candidate operations as an optimization problem for the architecture weight parameters in a continuous space so that the whole architecture search process can be differentiated to achieve gradient-based optimization, which is the most popular among the NAS algorithms at present. DARTS+[26] proposes introducing the early stopping mechanism in the search stage to address the issue of skip-connection enrichment during the training process of DARTS. This issue leads to a significant performance loss of the final model. Fair DARTS [12] also addresses the skip-connection enrichment phenomenon by transforming the candidate operations in the search phase from competition to cooperation. This is achieved by utilizing the Sigmoid function instead of Softmax to score the architectures. In addition, DARTS-[13] proposed using auxiliary skip-connections to leverage the advantages of skip-connections in search operations compared to other operations.

The objective of DARTS is to find a Normal Cell that maintains the output feature dimension and a Reduction Cell that reduces the output feature dimension by half. This is achieved by searching for the optimal combination of operations from the candidate operations in the search space. In the search phase, Cell can be viewed as a directed acyclic graph. The path from node $i$ to $j$ represents a candidate operation $o^{(i,j)}\in\mathcal{O}$, and node $j$ represents a potential feature mapping $x^{(j)}$. Each Cell comprises two input nodes, four intermediate nodes $x^{(j)}=\sum_{i<j}o^{(i,j)}(x^{(i)})$, and one output node. DARTS assigns an architectural weight vector $\alpha^{(i,j)}$ to each $o^{(i,j)}$, and relaxes the categorical choice of a particular operation by the output probability distribution of $Softmax$:

The task of Cell architecture search then reduces to learning a matrix $\alpha$ consisting of a set of continuous variables $\{\alpha^{(i,j)}\}$.

In contrast to the $\alpha$ weight-sharing strategy in DARTS, each Cell in Relax DARTS performs select operations based on a distinct weight matrix $\alpha$. Furthermore, the ratio of the two input nodes in the $Cell_{i}$, namely the inputs from $Cell_{i-1}$ and $Cell_{i-2}$, is adjusted by setting the learnable weight parameter $\beta_{i}=\{{\beta_{0}}^{i},{\beta_{1}}^{i}\}$. The inputs $s_{0}^{i}$ and $s_{1}^{i}$ of $Cell_{i}$ are represented as:

The objective of the search phase is to optimize the weights $\alpha$, $\beta$, and $w$ in the supernet, which is formed by all the mixing operations, using gradient descent. The training and validation loss, denoted by $\mathcal{L}_{train}$ and $\mathcal{L}_{val}$, respectively, are determined by the weights $\alpha$, $\beta$ and $w$ together. Specifically, architecture search aims to find $\alpha^{*}$ and $\beta^{*}$ corresponding to minimizing $\mathcal{L}_{val}$ by Eq.(4 and 5) and the corresponding network weights $w^{*}$ by minimizing $\mathcal{L}_{train}$ by Eq.(6). This task can be regarded as a triple optimization problem for joint optimization:

After obtaining the optimal cell structure by searching, training on the target task is restarted.

To minimize the performance loss caused by parameter $\alpha$ sharing and the stacking of identical cells in the network structure search, the local search strategy enables each cell to be endowed with an independent architectural parameter, thereby ensuring that each cell in the network is unique according to its current position. Specifically, we propose allowing each cell to update the structural parameter $\alpha$ independently. After initialization, $\alpha$ is used as the weight matrix for each cell’s node operation, enabling independent architecture selection. To reduce the risk of overfitting, we simplify the network structure by alternately stacking the Normal Cell and Reduction Cell 3 times to become a 6-layer network. At the end of the search phase, we obtained six distinct cells instead of two.

The performance gap that arises when using the searched network structure for training and evaluation has been a persistent issue due to differences in network architecture[27, 28]. To address this, we introduce the learnable parameter $\beta$ as the input weight of each cell to fine-tune the network architecture during training and optimize the global network structure. The $cell_{i}$ has two input feature vectors, $s^{i}_{0}$ and $s^{i}_{1}$. The input $s^{i}_{0}$ is the same as the input $s^{i-1}_{1}$ of the previous $cell_{i-1}$, which is a skip-connect operation at the network structure level. The output of the previous cell is the other input $s^{i}_{1}$, which is a direct-connect operation. The parameter $\beta$ is learnable and measures the proportion of cells that choose skip-connect and direct-connect operations. We use softmax for normalization after initializing $\beta$ in the same way as $\alpha$. The competition between the two inputs is preserved, and note that the proportions sum to 2 instead of 1, avoiding simultaneous scaling of the inputs. When the weights of the inputs are less than the threshold $c=0.2$, the inputs are replaced with the all-zero tensor.

A network structure can be determined by a set of parameters $\alpha,\beta$, and $\omega$. Evaluating the performance of the structure after optimizing the parameters at each step is costly. Therefore, we use the same approximation strategy as DARTS to perform an alternating triple optimization of $\alpha,\beta$ and $\omega$ without training each network to convergence:

The parameter optimization learning rate is represented by $\xi$, and Algorithm1 illustrates the overall search algorithm.

In detail, in the traditional deep learning approach, we select the first round of data for training and the second round of data collected at different times for testing. In the DARTS-based approach, the first round of data is first divided into a training set and a validation set for the automatic search of the network, and the resulting target network architecture is then utilized for training with the first round of data and testing with the second round of data.

Tables 1-3 list the EER and FRR@FAR of each approach. It is evident that our Relax DARTS outperforms existing approaches, achieving the lowest verification error, i.e. 0.0657, 0.0610, and 0.0529 on the three subsets of the GazeBase dataset, and 0.0437 on the JuDo1000 dataset. Also, it is clear that our proposed approach achieves higher recognition accuracy compared to existing approaches at different FARs.

To investigate the effect of each step on the model’s recognition accuracy improvement, we performed ablation experiments on the RAN sub-dataset in GazeBase. Specifically, we used DARTS as a baseline and added $\alpha$-independent optimization strategy, resulting in the model denoted as ‘+$\alpha$’. Finally, we introduced cell input weights $\beta$ for global network structure fine-tuning, resulting in the model expressed as ‘+$\beta$(Relax DARTS)’.

The verification error rates resulting from the ablation schemes are presented in Table5. The experimental results suggest that $\alpha$-independent optimization can significantly enhance the baseline model’s recognition performance. This is because the open $\alpha$-optimization strategy increases the degree of freedom of network search, resulting in unique cells that are adapted to the current network location. In addition, including input weights $\beta$ achieves the lowest EER and yields the best performance. This enables fine-tuning of the network architecture while the network performs cell searching, allowing for global search.