Learning Video-independent Eye Contact Segmentation from In-the-Wild Videos

For each frame $\bm{I}_{j}$ of the video $V_{m}$, we run an appearance-based gaze estimator to obtain the gaze vector $\bm{g}^{c}_{j}$ of the person of our interest in the camera coordinate system. Through the data normalization process for gaze estimation [59], we also obtain the center of the face $\bm{o}^{c}_{j}$ as the origin of the gaze vector. We also perform body pose estimation to obtain the translation vector $\bm{t}_{j}$ and the rotation matrix $\bm{R}_{j}$ from the body coordinate system to the camera coordinate system. $\bm{g}^{c}_{j}$ and $\bm{o}^{c}_{j}$ are then transformed from the camera coordinate system to the body coordinate system through $\bm{g}^{b}_{j}=\bm{R}_{j}^{-1}\bm{g}^{c}_{j}$ and $\bm{o}^{b}_{j}=\bm{R}_{j}^{-1}(\bm{o}^{c}_{j}-\bm{t}_{j})$, where $\bm{o}^{b}_{j}$ indicates the face center in the body coordinate system. Therefore, $\bm{o}^{b}_{j}+\beta\bm{g}^{b}_{j}$ defines the gaze ray in the body coordinate system.

For each video $V_{m}$, we then compute a set of intersection points $\{\bm{u}_{j}\}$ between the gaze rays and a cylinder centered at the origin of the body coordinate system with a radius of $r$ and convert these 3D intersection points to 2D gaze points $\{\bm{v}_{j}\}$ on the cylinder plane by cutting the cylinder along the line $(0,y,r)$ parameterized with $y$. We apply OPTICS clustering [1] on the set of 2D gaze points $\{\bm{v}_{j}\}$ and treat the identified clusters as eye contact regions for the $m$-th video $V_{m}$. We generate pseudo-labels $\{\bm{p}_{j}\}$ for $V_{m}$ by treating all identified eye contact regions as positive samples and others as negative samples.

We estimate the 3D body pose $\bm{R}$ and $\bm{t}$ of the target person based on the 3D model fitting. We define our six-point 3D generic body model according to the average human body statistics [34], consisting of nose, neck, left and right shoulder, and left and right waist keypoints. This 3D body model is in a right-handed coordinate system, which means that the chest-facing direction is the negative Z-axis direction. Given the corresponding 2D keypoint locations from the target image, we can fit the 3D model using the P6P algorithm [26] assuming (virtual) camera parameters. This gives us the translation vector $\bm{t}$ and the rotation matrix $\bm{R}$ from the body coordinate system to the camera coordinate system.

To locate the six 2D body keypoints, we rely on a pre-trained 2d keypoint-based pose estimator. For each frame, the pose estimator is expected to take the whole frame as input, and output body keypoints including the ones corresponding to our six-point body model. However, directly using the six 2D keypoints from the pose estimator can lead to inconsistent results throughout frames. This inconsistency arises from the fact that there normally exist at least two near-optimal solutions with similar reprojection errors due to the symmetric nature of the human body. Therefore, we introduce some subtle asymmetry in the 3D body model and stabilize the pose by introducing hard-coded keypoints, as illustrated in the lower part of Fig. 3. Specifically, we only use three keypoints that correspond to the left shoulder $\bm{s}^{l}$, the right shoulder $\bm{s}^{r}$, and the neck $\bm{n}$ from the pose estimator. We calculate the other three keypoints, the left waist $\bm{w}^{l}$, the right waist $\bm{w}^{r}$, and the nose $\bm{e}$, assuming that the target person is standing straight. The keypoints of the waist are defined as $\bm{w}^{l}=\bm{s}^{l}+(0,d)^{T}$ and $\bm{w}^{r}=\bm{s}^{r}+(0,d)^{T}$, where $d=|\bm{s}^{r}-\bm{s}^{l}|_{2}$ indicates the length of the shoulder, and the nose is defined as $\bm{e}=\bm{n}-(0,\alpha d)^{T}$. We also set the ratio $\alpha=1.632$ according to the same statistics of the human body [34].

The structure of the segmentation network is illustrated in the lower part of Fig. 4. For each frame in the input tracklet $T$ of length $I$, we extract and normalize the face image of the target person according to [59]. It is then fed to a pre-trained gaze estimation network based on the ResNet architecture [18] with 50 layers followed by a fully connected regression head. We extract the gaze feature vectors $\bm{f}_{i}\in\mathbb{R}^{2048}$ from the last layer and concatenate all gaze features $\{\bm{f}_{i}\}$ collected from the tracklet along the temporal dimension to form the gaze feature matrix $\bm{F}\in\mathbb{R}^{2048\times I}$, which will be used as input to the segmentation block.

Based on the MS-TCN++ architecture [28], the segmentation block consists of a prediction stage and several refinement stages stacked upon the prediction stage. The prediction stage has $11$ dual dilated layers. At the $l$-th dual dilated layer, the network performs two dilated convolutions with dilation rates $l$ and $11-l$. The features after the two dilated convolutions are first concatenated, so that the network is able to attend to long-range temporal information even at the early stage. This is then followed by ReLU activation, pointwise convolution and skip connections. The refinement stages are similar to the prediction stage, except that the 11 dual dilated layers are replaced with 10 dilated residual layers.

For the loss function, we follow the original paper and use a combination of cross-entropy loss $\mathcal{L}_{\textrm{cls}}$ and truncated mean squared loss $\mathcal{L}_{\textrm{t-mse}}$ [28]. $\mathcal{L}_{\textrm{t-mse}}$ is defined based on the difference in the prediction between adjacent frames and encourages smoother model prediction. The loss for a single stage is defined as $\mathcal{L}_{s}=\frac{1}{|B|}\sum_{b\in B}(\mathcal{L}_{\textrm{cls}}(\bm{p}_{b},\bm{y}_{b})+\lambda\mathcal{L}_{\textrm{t-mse}}(\bm{y}_{b}))$, where $B$ indicates the training batch, and $\lambda$ is a hyperparameter controlling the extent of $\mathcal{L}_{\textrm{t-mse}}$. Finally, the overall loss for all stages is the sum of $\mathcal{L}_{s}$ at each stage.

To extract tracklets, we run face detection and face recognition on each video in $\mathcal{V}$ in the training dataset. A tracklet $T_{k}$ is formed only if the IoU between the bounding boxes of the faces of consecutive frames is greater than $\tau_{\mathrm{IoU}}$. The tracklet is also disconnected if the neck and shoulder keypoints cannot be detected. We also discard short tracklets that do not exceed 4 seconds.

Since pseudo-labels extracted from the gaze target discovery are person-specific, we also need to make sure that the training tracklets are extracted from the specific target person. To this end, we add the cosine similarity threshold $\tau_{c}$ to construct training tracklets. Assuming that a set of reference face images is given, we compute the cosine similarity of the detected face with each of the reference faces. As long as one of them is greater than $\tau_{c}$, the tracklet continues. Note that this threshold is not required during inference.

Table. 1 shows the comparison of our proposed unsupervised video-independent eye contact segmentation model with unsupervised video-dependent eye contact detection baselines. The first row (CameraPlane + SVM) is the re-implementation of the unsupervised method of Zhang et al. [58] that extracts framewise pseudo-labels in the camera plane and trains an SVM-based eye contact classifier based on a single frame input. We did not set a safe margin around the positive cluster to filter out unconfident negative gaze points because we observed that it does not work well in in-the-wild videos, especially when there exist multiple gaze targets. The second row (CylinderPlane + SVM) applies our proposed cylinder plane gaze target discovery method, and an SVM is trained on the resulting pseudo-labels. Note that these two methods are video-dependent approaches, i.e., the models are trained specifically on the target video. Our proposed cylinder plane achieves $68.52\%$ accuracy in video-dependent eye contact detection, outperforming the camera plane baseline by $10.8\%$, indicating the advantage of our pseudo-label generator in in-the-wild videos. The last row (Ours) corresponds to the proposed unsupervised eye contact segmentation approach with an iterative training strategy. Our method achieves $71.88\%$ framewise accuracy, outperforming the video-dependent counterpart (CylinderPlane + SVM) by $3.36\%$ and the camera-plane baseline by $14.16\%$. It also achieves the highest segmented edit scores and F1 scores, indicating better segmentation qualities.

We also conduct ablation studies to show the effectiveness of our design choices, i.e, our problem formulation, our proposed gaze target discovery method and iterative training. The result is shown in Table. 2. CameraPlane + Generic SVM is the baseline method that obtains pseudo-labels for tracklets using the gaze target discovery method of Zhang et al. [58] and trains an SVM-based generic video-independent eye contact detector. We choose to use SVM as the classifier simply for comparison with video-dependent baseline approaches, and SVM is trained through online learning optimized by SGD. CylinderPlane + Generic SVM replaces the gaze target discovery method of Zhang et al. [58] with our proposed gaze target discovery method and, therefore, is also a video-independent eye contact detection approach. Although the baseline camera-plane approach outperforms our proposed cylinder-plane approach by $2.45\%$, both SVM-based detection models achieve framewise accuracy only slightly better than chance. CameraPlane + Ours obtains pseudo-labels on the camera plane but replaces the SVM with our segmentation model, making it a video-independent eye contact segmentation method. It outperforms its detection counterpart by $4.27\%$, showing the superiority of our problem formulation.

Our proposed method without iterative training (Ours (1 iteration)) achieves $70.04\%$ accuracy, showing the effectiveness of our proposed gaze target discovery when applied in the segmentation task setting. The last three rows of Table. 2 shows the effectiveness of the proposed iterative training strategy. From iteration $1$ to iteration $2$, we observed a decent improvement in model performance in terms of both accuracy and segmentation quality. There are also gradual model improvements even in subsequent iterations, indicating its effectiveness. Our proposed method in the last iteration outperforms that of the first iteration by $1.84\%$ in accuracy, and there is also a great improvement in the edit score and the F1 scores.

During both training and testing time, we use tracklets for more than 4 seconds as input to the model. In Table 3, we vary the length of the input tracklet during the test time and analyze its effect on the performance of the model. As can be seen, the model performance improves as we increase the input tracklet length. In particular, if the input length is more than one minute, our proposed method reaches an accuracy of $85.14\%$.

The higher performance observed in longer tracklets can also be attributed to the performance of the model in different head poses. During the tracklet formation stage, we use face recognition to filter celebrity faces. Since face recognition performance is limited on profile faces, tracklets containing extreme head poses tend to be much shorter than those containing only frontal faces. During training, the lack of long tracklets with profile faces prevents the model from modeling long-term eye contact dependencies, leading to degraded performance on tracklets that contain mainly profile faces. The longer tracklets in the test set also contain mainly frontal faces, resulting in higher accuracy.

In Fig. 6(a), we visualize the accuracy of our trained model conditioned on different head poses. We use HopeNet [42] to extract head poses of celebrity faces in each frame of the test tracklets and compute framewise accuracy for all yaw-pitch intervals. We can observe that our model works the best when the pitch is between $-40$ and $-60$ degrees, i.e., looking downward. It can also achieve decent performance when both yaw and pitch are around $0$ degrees. However, when the yaw is lower than $-60$ degrees or higher than $60$ degrees, our model is even worse than random chance.

In Fig. 6(b), we show some qualitative visualizations. We randomly present test tracklets with frontal faces in the first row and test tracklets with profile faces in the second. Although our model has decent performance on frontal-face tracklets, the predictions on profile-face tracklets seem almost random. We also present test tracklets longer than one minute in the third row, and all of these long tracks contain frontal faces. Consequently, we can observe mainly fine-grained eye contact predictions in these examples.

Another limitation of our proposed approach lies in our full-face appearance-based estimator. Fundamentally, the line of work on full-face appearance-based gaze estimation regresses the face images into gaze directions in the normalized camera coordinate system. Consequently, the gaze features used as input to the segmentation model only have semantic meaning in the normalized space, but not in the real camera space. In addition, the full-face appearance-based estimator trained on the gaze dataset collected in controlled settings tends to have limited performance in unconstrained images, especially when the person has extreme head poses unseen in the training dataset. This may also be a reason for the low performance of the model on the face of the profile.

Finally, our approach cannot handle the cases of moving gaze targets and humans. Our gaze target discovery assumed fixed relative positions between the eye contact target and the person and would consequently give incorrect pseudo-labels in such cases. Eye contact detection with moving gaze targets is a more challenging task than that with stable gaze targets. We argue that in this case gaze information will not be sufficient for the model. Information about the spatial relationship between the person and gaze targets should be introduced.