FlatTrack: Eye-tracking with ultra-thin lensless cameras

Eye tracking plays a crucial role in Augmented and Virtual Reality (AR/VR) systems, whether it be augmenting the realism and interactivity of the immersive experience or rendering efficient display imagery. For example, by precisely tracking where the user is looking at any given time, eye tracking facilitates more natural and intuitive interactions, such as gaze-based interfaces for selection and control, which can enhance the user engagement with the virtual environment without the need for physical controllers [20, 24]. Eye tracking also facilitates optimized usage of system resources by rendering high-resolution graphics only in the regions of the visual field where the user’s gaze is well-focused, and there is less detail in the periphery, a technique called foveated rendering that is shown to dramatically improve the display performance[12, 1, 19, 9]. Apart from improving the overall graphical performance through selective on-demand rendering, foveated rendering significantly reduces computational load, allowing for smoother and more immersive AR/VR experiences.

Existing eye-tracking technologies (Tobii Pro, Pupil Labs) face several significant challenges that impact their practicality across various applications despite their rapid developments in recent years. Apart from the need for more accuracy and reliability, the physical size of eye-tracking units often limits their integration into wearable AR/VR headsets, posing significant design issues. Specifically, the optical components like lenses, filters and their housing directly affect the overall size of the device. The structure used to mount the sensors and associated electronics, including the enclosure that protects these components, further adds to the bulk of the system. Furthermore, privacy concerns also arise as eye tracking can reveal sensitive personal information. As demonstrated in earlier works, eye images can possibly be misused to reconstruct the scenes being observed, which pose significant threats and privacy concerns[2]. Finally, eye-trackers require low latency to ensure responsive and immersive user experiences in interactive applications. These obstacles necessitate continuous advancements in hardware and software to enhance the versatility and user-friendliness of eye-tracking technology. Hence, we propose a lensless gaze-tracking prototype that has a small form factor, inexpensive, lightweight, and privacy-preserving[14, 7].

In lensless imaging, the focusing lens is replaced with a thin, lightweight, and potentially inexpensive optical encoder, resulting in flat and lightweight miniaturized cameras. A lensless camera uses advanced computational reconstruction algorithms to recover the scene from captured sensor measurements, which no longer resemble the imaged scene [7, 4, 5]. However, with a well-designed encoder, the measurements contain sufficient information to recover an image of the scene using post-processing algorithms that can demultiplex the sensor measurements to reconstruct a sharp image of the scene. While digital post-processing of images is standard for tasks such as distortion correction, synthetic depth of field, and denoising, lensless cameras are fundamentally different in that the post-processing is a part of the imaging system design where the optical imaging hardware, and the algorithmic software is designed together. Moreover, since lensless cameras encode information indirectly in the measurement, then computationally extract it by solving an inverse problem, they also provide promise for enabling privacy in the eye tracking measurements[14]. Additionally, lensless cameras inherently capture the 3D information, which ensures a more accurate prediction[6].

In the current work, we employ a lensless camera design accompanied by a lightweight convolutional neural network to achieve compact eye tracking. Specifically, we use a Near-Infrared (NIR) PhlatCam[7] to develop an ultra-thin near-eye tracker. Using our system, we collect the first-of-its-kind large dataset of nearly 20000 paired lensless captures and calibrated gaze vectors. We propose a two-stage approach for estimating gaze from lensless capture and evaluate the efficacy of such an approach on our proposed dataset. Our experiments show that lensless cameras allow us to build compact gaze trackers with high-fidelity gaze estimates. We plan to release the dataset and gaze tracking algorithm upon acceptance of the paper.

In summary, our contributions in this paper are:

• •

  We build a system using NIR PhlatCam prototype for gaze tracking. Existing works on lensless gaze estimation have mostly shown results on simulated data that don’t capture the real data’s non-idealities.

• •

  We collect a large dataset of 20475 paired lensless captures and gaze directions from a near-eye setup. To the best of our knowledge, we are the first to collect such a dataset for lensless gaze tracking.

• •

  We build a two-stage gaze estimator to evaluate the efficacy of lensless gaze estimation and demonstrate that lensless cameras allow high-fidelity gaze recovery with a much smaller form factor. Our gaze regressor performs real-time ($>$125 fps) gaze estimation.

Mask-based Lensless Imaging. Mask-based lensless cameras replace the lens of conventional cameras with a thin optical mask. This mask has the ability to modulate the incoming light and can be placed extremely close to the sensor, leading to a flat form factor. Moreover, the ultra-small mask-to-sensor distance allows these cameras to have a large depth of field, thereby making it possible to image from a very close range - a property that will allow lensless camera-based gaze trackers to have a much smaller form factor. Multiple lensless cameras have been developed in the literature. FlatCam [5] is a lensless camera that places a separable coded amplitude mask above a bare sensor array to enable a thin and flat form-factor imaging device, which can simulate a conventional camera by reconstructing conventional images from coded measurements. DiffuserCam [4] and PhlatCam [7] replace the coded amplitude mask from FlatCam [5] with a coded phase mask for improved light efficiency and reconstruction quality. Spectral DiffuserCam[17] exploits the multiplexing ability of lensless imagers to do hyper-spectral imaging. The primary differences in the above design lie in the pattern of the mask used and the mask-to-sensor distance. In this work, we will primarily focus on the PhlatCam [7] lensless camera. Previous works like [6, 14] have shown the application of this camera design for 3D imaging and optical encryption. Previously, FlatCam [5] was shown in [23] to perform at par with lensed cameras for gaze-tracking. However, the authors only reported the performance on simulated data, and ensuring the same performance on real data is hard. We take a step further and develop the first-of-its-kind real lensless gaze dataset using PhlatCam. We run extensive experiments on this real dataset to highlight the efficacy of our system. We plan to release this dataset upon acceptance of the paper.

Gaze Tracking Datasets. Since learning-based gaze trackers need large datasets, many such datasets have been proposed in the literature for both remote and near-eye gaze tracking. Among the remote gaze tracking datasets, MPII-Gaze[25] contains 214K face images captured using a laptop camera from 15 subjects along with their gaze vectors. Columbia Gaze[22] contains 5880 images from 56 people and focused on gaze locking. GazeCapture[16] contains 2.5M frames collected from over 1450 people. Among the near-eye gaze tracking datasets, NVGaze[15] contains both real and synthetic data of 2M infrared images of eyes at 1280 × 960 resolution. OpenEDS[18] contains more than 500K eye-images with the corresponding 3D gaze vectors collected from 80 participants. Recently, [3] proposed a dataset captured using a DAVIS event camera. This dataset contains both event and grayscale frames along with the corresponding gaze. Although the existing datasets contain a wide distribution of eye images and environment, none of them are suited for lensless cameras, which have a very distinct imaging model. Thus, there’s an urgent need for a dataset developed specifically for lensless cameras.

In a mask-based lensless camera[5, 7] like the one used in this work, the conventional lens is replaced by a thin optical mask placed at some distance from the sensor that modulates the incoming light. For a scene $X$, the measurement recorded by a sufficiently large sensor $Y$ is given by:

$$Y=P*X+N,$$

(1)

where $*$ is the full-size convolutional operator (no cropping due to finite sensor size), $P$ is the point spread function (PSF), and $N$ is additive noise. The PSF is the response of the camera to a point source. The optical mask can be implemented using an amplitude mask that attenuates the incoming light [5] or a phase mask that modulates based on diffraction [7]. In this work, we use a phase mask. The lensless PSF is a function of the mask pattern, wavelength of light, and the mask-sensor distance. Our PhlatCam uses a phase mask designed for NIR wavelength of 700nm and is placed slightly less than 1.5mm from the sensor.

Due to the lack of a focusing element like a lens, a lensless camera capture is a multiplexed representation of the scene. To recover the original scene, one needs to computationally solve an inverse problem of estimating the scene $X$ given the measurement $Y$ and PSF $P$, which is challenging due to properties like large PSF support (which can be bigger than the scene projection $X$), finite sensor size, noise, etc. Although there exist different methods of lensless scene reconstruction in the literature, we will primarily focus on the two most popular methods – FlatNet[13] and Wiener deconvolution. Please see Fig. 2 for the lensless imaging and reconstruction process.

We use an NIR-PhlatCam designed for 700nm to capture the FlatTrack Gaze Dataset. We collected a comprehensive dataset comprising 20,475 lensless measurements meticulously gathered from 13 distinct subjects. The importance of user-specific data lies in the practical application of eye-tracking systems, where such systems are typically fine-tuned or calibrated for individual users using a small amount of personalized data. This calibration process is essential in real-world scenarios, such as AR/VR applications, where precise gaze estimation requires personalization.

Each subject participated in a series of experiments in which they were tasked with focusing on stimuli presented on a computer screen. The stimuli moved along a 15x15 grid pattern as in Fig. 3 within a 1080x1920 pixel area of the monitor (DELL S2421HN), and their eye images were captured by the lensless camera. To ensure accuracy, subjects were seated 50 cm away from the screen, and before each round of data collection, calibration was performed to align the center of the eye with the origin on the screen. This process was repeated for each round of data collection for every subject. We also ensured that the distance between the eye of the subject and PhlatCam is approximately 4cm. We collected 4 to 5 rounds of data for each subject, each round comprising either 225 or 675 images (multiple measurements) corresponding to the 15 x 15 grid pattern. For each measurement, we also note the gaze point in pixels on the monitor and convert it to a unit vector in the gaze direction using the known monitor distance.

The distance between two consecutive stimulus points along the x-direction in the grid pattern measures 121.3 pixels, while along the y-axis, it is 66.3 pixels. A margin of 50 pixels was also uniformly removed from all four sides of the grid to mitigate any potential edge effects. The field of view (FoV) subtended by the eye along the x-axis, spans 53.03 degrees. Similarly, the FoV along the y-axis extends to 29.6 degrees. Our capture setup, grid pattern, and lensless reconstruction of captured data are shown in Fig. 3.

Analysis of the angular differences between two consecutive grid points along the x-axis reveals a distinct trend characterized by an initial increase, peaking, and subsequent reduction. The minimum angle along the x-axis and y-axis is observed to be 3.21 degrees and 1.77 degrees, respectively. These trends are observed along the edges as the angular distance reduces with distance from the eye.

Since lensless measurements are globally multiplexed and do not resemble the scene, we use a two-stage approach: in the first stage, we use a reconstruction algorithm to obtain the scene estimate from the lensless captures; in the second stage, we use a gaze-regressor neural network to obtain the gaze direction. We discuss the approach in detail below.

Scene Reconstruction. Given the lensless forward model described in Eq. 1 Scene reconstruction from lensless capture involves solving an inverse problem of estimating the scene $X$, given the measurement $Y$ and PSF $P$, i.e.,

$$\hat{X}=\mathcal{F}(Y,P),$$

(2)

where $\hat{X}$ is the scene estimate and $\mathcal{F}$ is the reconstruction algorithm. In this work, we experimented with two different approaches for $\mathcal{F}(.)$ – Tikhonov-regularized least-squares (also known as Wiener deconvolution) and a learning-based algorithm called FlatNet[13]. Tikhonov Regularization involves solving the following equation, where Y is the lensless measurement, P is the PSF, and $X_{T}$ is the scene estimate.

$$\hat{X}_{T}=\arg\min_{X}\left\|Y-P*X\right\|_{F}^{2}+\gamma\left\|X\right\|_{F}^{2}$$

(3)

. This equation has the following closed-form solution, where $F(.)$ is the Fourier transform and gamma is set to 1e-5.

$$\hat{X_{T}}=F^{-1}\left(\frac{(F(P))^{*}\odot F(Y)}{|F(P)|^{2}+\gamma}\right)$$

(4)

For the learning-based FlatNet, we use pre-trained weights obtained by training it on a natural image dataset. FlatNet consists of two stages: (1) an inversion stage that maps the measurement into a space of intermediate reconstruction by learning parameters within the forward model formulation, and (2) a perceptual enhancement stage that improves the perceptual quality of this intermediate reconstruction. These stages are trained together in an end-to-end manner.

Gaze Regressor. Once the scene estimate $\hat{X}$ is obtained, we use a gaze regression network to finally predict the gaze direction. Our network first predicts the unit vector corresponding to the gaze direction. We then project this 3D gaze direction to a 2D gaze point (in pixels) on the monitor using the known distance between the monitor and the eye.

Loss Function. Once the gaze points on the monitor are estimated, we then use L1 loss on the predicted and groundtruth gaze points. More specifically, we use,

$$\mathcal{L}=||G_{GT}-G_{PRED}||_{1},$$

(5)

where $G_{GT}$ and $G_{PRED}$ are the ground-truth gaze point and predicted gaze point, respectively. Finally, we use this loss to update the gaze regression stage while keeping the reconstruction stage frozen. We show our gaze estimation pipeline in Fig. 4.

In this section, we provide details of the implementation of our training strategy and a comprehensive evaluation of the gaze estimator. We benchmark various eye-tracking algorithms on our FlatTrack dataset and evaluate the performance on angular error and inference times. We also perform a quantitative comparison of lens-based v/s lensless imaging systems. We also assess the different lensless reconstruction methods: Wiener and FlatNet. Finally, we analyze the average per-pixel error of gaze estimation.

We propose a gaze-tracking framework for thin, lensless cameras. Lensless cameras, due to their ultra-thin form factor and lightweight nature, are attractive for applications requiring eye gaze tracking, such as Augmented/Virtual Reality. However, existing works on gaze estimations have not fully exploited this advantage. One of the bottlenecks for fully exploiting the benefits of lensless cameras for gaze tracking is the lack of a lensless gaze dataset. To deal with this, we propose the FlatTrack dataset - a first-of-its-kind NIR PhlatCam gaze dataset collected from 13 subjects and contains around 20K paired lensless measurements and gaze ground truth. Using a custom lensless gaze estimation pipeline evaluated on the FlatTrack dataset, we show that lensless gaze estimates can be of high fidelity. Furthermore, these gaze estimation algorithms can run at very high speeds ($>$125 fps) on a typical GPU. We hope the proposed dataset and the pipeline, when released, will drive more innovation in this promising space.

Acknowledgments: KM acknowledges funding support from IITM Pravartak Technologies Foundation, Department of Science and Technology, India (CRG/2023/007358) and Qualcomm Faculty Award 2024.