Hand-Priming in Object Localization for Assistive Egocentric Vision

Interactions between hands and objects in egocentric vision have been explored for various tasks: from gaze estimation to human behavior understanding. The estimation of hand pose and shape can serve as a cue to understanding users’ intentions and thus has been the focus of research for both the computer vision and human-computer interaction communities [45, 9, 26, 51].

Traditionally, the egocentric hand information has been utilized in activity recognition [49, 15, 40, 33]. Prior work focuses on the interactions between hands and objects due to that objects can possess a clue to users’ activities [49, 40, 31, 33]. Using two-stream convolutional neural networks, Ma and Kris [33] try to localize and recognize an object of interest according to the hand pose and location in egocentric vision to recognize users’ activities. This prior work suggests localizing the center area of the object of interest, instead of its exact center coordinates. Our method also follows this approach for object localization.

Some prior work assumes that some implicit visual attention, such as gaze information, has been implicitly encoded in input data; hence, there is prior work that tries to estimate gaze points in egocentric data [28, 29, 21, 50] or uses this visual attention to accomplish other tasks [14, 31, 63]. Such visual attention, however, may not be encoded in data from different populations — for instance, blind people. For this issue, recent prior work has proposed to use more explicit visual attention, such as fingers pointing to the item of interest, to understand items of interest of blind people in complex visual scenes [18]. In addition, even for people with visual impairments, hands have been found to be the explicit cue to objects of their interest [23, 26]. Based on these prior observations, our work focuses on interactions between egocentric hands and objects of interest to learn the relationship between the hands and objects of interest.

Hand–object interactions have also been explored for other tasks. Tekin et al. [51] propose a single neural network model that outputs 3D hand and object poses from RGB images and recognizes objects and users’ actions. Cai et al. [7] suggest to model the relationship between hand poses and object attributes, as those data can provide complementary information to each other. With the model that understands the hand–object relationship, this prior work further tries to recognize users’ actions. As our model localizes an object of interest based on the hand–object interactions, we see its potential usage in other applications, such as recognition of objects and actions.

Prior work in cognitive science has shown that the human visual system tries to use “context” to pay attention only to our interest [53, 57, 34]. The use of the context information has also been discussed from the perspective of computer vision [38, 37, 13]. Inspired by these observations, researchers in computer vision have introduced various approaches that exploit contextual information [52, 36, 42, 58, 35].

In the deep learning era, researchers have also shown that using contextual information, such as instance segmentation, helps to improve the performance of object detection and recognition [62, 47, 30, 20]. FCIS [30] and Mask R-CNN [20] are designed to learn instance segmentation, which is then used for object classification. On the other hand, Shrivastava and Gupta [47] proposed a slightly different approach. In their model, a segmentation module is used to provide contextual feedback in the object detection model (Faster R-CNN [43]); i.e., the segmentation output is appended to the object detection model. Built upon this contextual priming and feedback approach, our approach takes into account the contextual information (i.e., the hand information) for object localization in egocentric vision.

Assistive technologies, especially for people with visual impairments, have employed computer vision algorithms to help them access visual information that surrounds them. As people with visual impairments use their smartphones or wearable cameras to take photos of their target, the captured data are in egocentric vision; thus, assistive technologies have focused on computer vision models that can understand the egocentric vision. For instance, researchers in assistive technologies have proposed various approaches that can help visually impaired people capture their surroundings and access to that information. Such technologies, called blind photography, have been developed to simply capture scenes [54, 55], and recognize objects [5, 56, 22, 61] or people (e.g., family members and friends) [56, 1, 60]. In particular, prior work on blind photography reported that people with visual impairments are vulnerable to false-positive object localization [25] as there are no means for these users to check the localization output. In this paper, considering this prior finding, we focus on reducing false positives in object localization and evaluating the object localization models with the precision metric as this measurement captures false-positive estimations.

We build an object localization model that is reinforced by a hand segmentation output. Our hand-primed object localization model consists of two models: a hand segmentation network and an object localization network. For both of the models, we employ the FCN-8s architecture [32]. Using the pixel-level classification model, our object localization model tries to estimate which pixels belong to the center area of an object of interest as proposed in [33]; the one proposed in this prior work is used as the baseline model that uses the hand information for object localization.

As depicted in Figure 2, the hand segmentation output is appended to several layers in the object localization network to prime the model to use the hand information for object localization. We propose two different ways of priming the object localization model to use the hand information: HandPrimedAll (HPAll) and HandPrimedLate (HPLate). Assuming that providing this segmentation to all layers would make a model primed to use that feedback for its tasks [47], we design the HPAll model in which the segmentation output is infused to all the convolutional layers (conv1–conv5). On the other hand, in the HPLate model, the hand segmentation output is infused into the later convolutional layers (conv4, conv5). This approach is inspired by the observations that the later convolutional layers capture the more descriptive features [59], and the hand features are found to be captured in a later convolutional layer (conv5) of the prior object localization network [33].

The hand segmentation model has two output layers to segment hands and a background in an image; i.e., the model performs a per-pixel classification to determine which pixels belong to the hand or the background. For object localization, our model does not estimate the exact center and bounding box of an object of interest. Instead, the model infers the center area of a target object. Having two output layers, our hand-primed object localization model determines which pixels belong to the center area of an object of interest or the non-center area (background). Prior work in pose estimation [39] and egocentric activity recognition [33] has highlighted benefits of estimating areas of interest over the exact coordinate estimation.

Our model expects egocentric images that are obtained by users taking photography or by extracting salient frames from videos recorded using a wearable camera on the head, the chest, or eyeglasses.

As the object localization output is dependent on the hand segmentation output, we separately trained these two networks. First, we trained the hand segmentation network. Then, while freezing the weights of the hand segmentation network, we trained the object localization model.

Adam [24] was used to train our hand segmentation and object localization networks. Following are hyper-parameters that we set for training: (hand segmentation) 10,000 training steps, 0.00001 learning rate, 16 batch size, and $10^{-9}$ Adam’s epsilon; (object localization) 20,000 training steps, 0.00001 learning rate, 8 batch size, and $10^{-9}$ Adam’s epsilon. In both of the models, we initialized the weights of the first five convolutional layers (conv1–conv5) with those of the VGG-16 network model [48] pre-trained on ImageNet [12].

Training:

Using four egocentric datasets including EgoHands [2], GTEA [15], GTEA Gaze+ [14], and TEgO [26], we first trained our hand segmentation model. Provided with a set of original images and those hand masks as shown in Figure 3, our hand segmentation model learns hand features from the input data during training. To train our object localization model, a set of original images and those object center annotation data were used; Figure 3 shows the examples. The center of an object of interest in each image was annotated with a Gaussian heatmap blob. The object localization model was taught to estimate the center area of a target object interacted with the hand; i.e., we trained the model to learn the relationship between the hand pose and object location. GTEA, GTEA Gaze+, and TEgO were used to train and test the object localization model. Note that we did not use the EgoHands dataset for object localization as this dataset mostly contains hand movements in board games, such as chess, card games, and Jenga. As both of the hand segmentation and object localization models predict a class per pixel, the cross-entropy loss function was used to train both of the networks.

Inference:

At test time for object localization, we only considered per-pixel classification outputs which confidence scores were higher than 0.5 in the object localization output layer. When there was more than one cluster of the estimated center area, the biggest cluster was chosen to be the estimated center area.

All the datasets (GTEA, GTEA Gaze+, TEgO) already provide hand segmentation data for their original images, but only the TEgO dataset provides the annotation data for object location; they used a Gaussian heatmap blob to indicate the center of an object of interest, which was also used for egocentric activity recognition [33]. For the other datasets (GTEA and GTEA Gaze+), we used the object localization annotation provided by Lee and Kacorri [26]; they shared their manual annotation data for these datasets on the TEgO website. Following is the number of images used from each dataset: GTEA (663), GTEA Gaze+ (1,115), and TEgO (5,758). Note that we excluded images without hands from the TEgO dataset in our evaluation. We randomly selected samples from each dataset and applied the following dataset splitting to each dataset: 80% for training, 10% for validation, and 10% for testing.

The GTEA datasets include images only collected by sighted people, but the TEgO dataset contains data from a blind person. The evaluation with this TEgO data would gauge the compatibility of each localization model with assistive technologies for people with visual impairments, such as blind photography or object recognition.

There were four different models with which we used to compare our approach. Similar to our model, all the comparison models were also built upon the FCN-8s model. As in our method, the Adam optimizer was used for training.

NoHand:

This model was trained¹¹110,000 training steps, 0.00001 learning rate, 16 batch size. for object localization without explicitly being trained on hand segmentation data; i.e., the object center annotation was only used for training.

Finetune:

This is the baseline model that uses hand features for object localization, which was introduced by Ma et al. [33]. To implement this model, we fine-tuned\@footnotemark our hand segmentation model to the object localization problem for each dataset. During the fine-tuning, we froze the weights of the first five convolutional layers (conv1–conv5) as in the model proposed by the prior work; they observed that hand features were captured in the conv5 layer.

MultiClass/MultiClass-2x:

In this model, learning the hand and the object center was considered as a multi-class problem. This multi-class model estimates the class of each pixel; i.e., it determines to which class among background, hand, and object center area each pixel belongs. In addition, we implemented another multi-class model, MultiClass-2x, in which we set twice more weight on the loss in object localization. For both of the models, the weights were also initialized with those of our object hand segmentation model before training\@footnotemark.

MultiTask/MultiTask-2x:

Considering hand segmentation and object localization as two different tasks in one model, we employed the hard parameter sharing for this multi-task learning [8]. This model shares the first five convolutional layers (conv1–conv5) and has two separate branches — one branch for the hand segmentation and another for the object localization. This model is called MultiTask. Moreover, as in the MultiClass models, we also created another multi-task model in which the loss in object localization model was computed to be twice more important than the loss in hand segmentation. We call this model MultiTask-2x. Before training²²220,000 training steps, 0.00001 learning rate, 8 batch size. these models, we also initialized the weights of the hand segmentation part of these models with those of our hand segmentation model; i.e., the weights of the shared convolutional layers and the separate layers for the hand segmentation part were initialized with those of our hand segmentation model.

We first evaluate the hand segmentation performance of the models as our work focuses on this information to localize an object of interest in egocentric vision. A threshold of $0.5$ was used to determine the per-pixel classification on the hand segmentation output layer. We measured the hand segmentation performance of Finetune before fine-tuning it to object localization; thus, three models (Finetune, HPAll, and HPLate) ended up having the same hand segmentation model. Note that the hand segmentation part for the multi-class (MultiClass, MultiClass-2x) and multi-task (MultiTask, MultiTask-2x) models was initialized with the weights of our hand segmentation model and then fine-tuned to each dataset again.

Quantitative analysis:

We report the quantitative analysis of the hand segmentation performance of each model in Table 1. Note that NoHand was not evaluated for hand segmentation since this model was trained only for object localization. As described in the table, our hand segmentation model and the multi-task models achieved equal to or higher than 0.9 in mIoU and 0.85 in AP in all of the three datasets. On the other hand, the hand segmentation performances of the multi-class models were not competitive to the other models including our hand segmentation model. In particular, although the performance of the localization-focused multi-task model (MultiTask-2x) showed the similar performance to that of the naive multi-task model (MultiTask), we observed the performance degradation in the localization-focused multi-class model (MultiClass-2x) when comparing its performance to that of the naive multi-class model (MultiClass) in all of the datasets.

Qualitative analysis:

The hand segmentation outputs of all the models (except for NoHand) are visualized with the green color in Figure 4. As in the quantitative analysis, our visual inspection informs that all the models were able to segment hand(s) from testing images, but the multi-class models lack detailed segmentation. In general, the multi-class models segmented a large part of hand(s) from the testing images, but sometimes were unable to provide fine-grained hand segmentation. In the figure, we observed that the multi-class models were unable to segment fingers from the testing examples from GTEA and GTEA Gaze+.

As in the hand segmentation analysis, we evaluate the object localization performance, quantitatively and qualitatively. A threshold of $0.5$ was used to determine whether per-pixel estimations belong to the center area of an object.

Quantitative analysis:

Table 2 shows the object localization performances of our hand-primed object localization models (HPAll, HPLate) and the comparison models. In the GTEA and TEgO datasets, our model (HPLate) outperformed the comparison models in almost all of the metrics. In the GTEA Gaze+ dataset, our HPLate model showed the best performance in $AP_{75}$ and comparative performances to those of MultiClass-2x in the other metrics, such as $mIoU$, $AP$, and $AP_{50}$.

Between our hand-primed object localization models, HPLate generally showed better performance than HPAll in all of the datasets, which indicates that modulation of hand features in higher layers helps capture the contextual relationship between hands and the object of interest more than modulating hand features in all layers. Compared with the baseline model (Finetune), our model (HPLate) achieved the performance improvement by 17% (from 0.47 to 0.55 AP), 50% (from 0.2 to 0.3 AP), and 12% (from 0.49 to 0.55 AP) in GTEA, GTEA Gaze+, and TEgo datasets, respectively. This result demonstrates that the explicit use (infusing) of the hand segmentation in object localization is more helpful for localizing an object of interest than fine-tuning the fixed features of the hand model to a different problem, object localization. Furthermore, in comparison to NoHand that did not have the explicit training for hand information, our model (HPLate) showed better object localization performance in all the three datasets. Based on these observations, it seems that having explicit hand features in the last convolutional layers (e.g., conv4 and conv5) helps to prime the object localization model to focus on necessary features, such as hand pose and location.

Qualitative analysis:

Figure 5 shows that our hand-priming models (HPAll, HPLate) well predicted the center area of an object associated with hand(s). As described in the quantitative analysis, HPLate generated more precise localization outputs than HPAll; the testing examples depict that HPAll tended to output a larger center area than HPLate and sometimes estimated more than one center area as shown in the GTEA Gaze+ example. MultiClass and MultiTask were unable to estimate the center area of an object of interest successfully, while the localization-focused version of these models (MultiClass-2x, MultiTask-2x) improved the performance of object localization.

More testing examples in Figure 6 were used to inspect the performances of the models, visually. The figure shows that our model estimated a more round shape of the center area, which is closer to the ground truth, than did the other models. In particular, comparing the output of our method (HPLate) with that of NoHand, we found further evidence that the hand information is essential not only to localize an object of interest but also to decide which object would be of interest when multiple objects appear in the scenes.

Failure cases:

Figure 7 shows failures cases of our model (HPLate) in object localization. It can still be confused with two or more objects when the hand–object interaction is not so obvious. For example, the left example in the figure shows that our model was confused with the two objects in proximity of the hand. This confusion seems to be caused by an indiscernible interaction between the hand and the object in the example. In addition, the middle example shows the false-negative localization of our model. Although our hand model successfully segmented the hand, the object model was unable to localize the object being held by the hand in the example. Furthermore, the false-positive output of our model is presented on the right of the figure. Despite the appropriate hand segmentation, our model localized a wrong object, which might be caused by the object occlusion by the hand. Further investigations are necessary to spot factors in these issues since the other methods also suffered from the issues in the same and/or different examples.

Dataset	Model	$mIoU$	$AP$	$AP_{50}$	$AP_{75}$
TEgO (blind)	NoHand	0.71	0.52	0.90	0.54
	Finetune	0.71	0.51	0.90	0.52
	MultiClass	0.48	0.13	0.54	0.01
	MultiClass-2x	0.75	0.57	0.93	0.61
	MultiTask	0.31	0.03	0.17	0
	MultiTask-2x	0.71	0.50	0.91	0.50
	HPAll	0.69	0.46	0.86	0.45
	HPLate	0.75	0.57	0.94	0.64
TEgO (sighted)	NoHand	0.74	0.54	0.93	0.58
	Finetune	0.74	0.55	0.96	0.57
	MultiClass	0.23	0.01	0.05	0
	MultiClass-2x	0.73	0.52	0.94	0.51
	MultiTask	0.52	0.17	0.63	0.03
	MultiTask-2x	0.74	0.53	0.96	0.54
	HPAll	0.73	0.51	0.94	0.50
	HPLate	0.76	0.59	0.96	0.64

More in TEgO:

Data from blind people may not have some implicit information about the object of interest (e.g., placement of the target object at the center of the camera frame). Hence, in data from blind people, hands may be a reliable indication of a target object. As only TEgO includes data collected by a blind person, we evaluated all the methods with the TEgO data collected by the blind person. In Table 3, we report the performance of each model on data from sighted and blind people in the TEgO dataset, respectively. For this analysis, the data from the sighted person and the data from the blind person were separately used to train and test each model. Our method, HPLate, outperformed the other methods not only on TEgO (sighted) but also on TEgO (blind). In particular, compared to the baseline model (Finetune), HPLate achieves the performance improvement, on average, by 12% and 7% on the TEgO (blind) and TEgO (sighted) datasets, respectively.