Target-absent Human Attention

To capture the information a person acquires from an image through a sequence of fixations, we propose a novel state representation, called Foveated Feature Map (FFMs). Fig. 1 shows an overview of how our FFMs are constructed. FFMs take advantage of pretrained ConvNets, which produce a pyramid of feature maps with progressively larger receptive fields. By treating deeper feature maps as information obtained at larger eccentricity (lower-resolution) in the peripheral vision, we construct FFMs as a set of multi-resolution feature maps, which is a weighted combination of different levels of feature maps using foveated weight maps generated based on previous fixations. Similar to image foveation [44], in FFMs, deeper feature maps with a lower resolution (corresponding to a larger eccentricity) are more weighted at locations with increasing distance from the fixation points. Below we discuss FFMs in greater detail.

For multiple fixations $\{f_{1},\cdots,f_{n}\}$, we compute the combined resolution map by taking the maximum at every location: $R(x,y|\{f_{1},\cdots,f_{n}\})=\max_{i}R(x,y|f_{i})$. In contrast to [34], which creates a Gaussian pyramid of the given image $I$ to produce the multi-resolutional version of $I$, we take inspiration from the Feature Pyramid Network [28] and use the in-network feature pyramid produced by existing pretrained ConvNets and blend the feature maps at each level of the feature pyramid to construct multi-resolutional feature maps (i.e., FFMs) based on the relative resolution map $R(x,y|f)$. For brevity, we will write $R(x,y|f)$ as $R(x,y)$ in the following text.

Each level of the feature pyramid $P_{i}$ represents a certain eccentricity, corresponding to a fixed spatial resolution, which we denote as $R_{i}^{*}$. It is defined as the relative resolution where a transfer function $T_{i}(\cdot)$ is at its half maximum, i.e., $T_{i}(R_{i}^{*})=0.5$ [34]. The transfer function $T_{i}(\cdot)$ is the function that maps relative resolution $r$ to relative amplitude, and it is defined as:

It can be shown that $R_{1}^{*}>R_{2}^{*}>R_{3}^{*}>R_{4}^{*}>R_{5}^{*}$, forming four resolution bins whose boundaries are defined by $R_{i}^{*}$ and $R_{i-1}^{*}$ ($i\in\{2,3,4,5\}$). To compute the weights at location $(x,y)$, we first determine which bin pixel $(x,y)$ falls in, according to its relative resolution $R(x,y)$ (see the supplementary material for more details). Assume pixel $(x,y)$ falls in between layer $j$ and $j-1$, i.e., $R_{j-1}^{*}\geq R(x,y)>R_{j}^{*}$. Then, we set the weights at layer $j$ and $j-1$ to be the ratio of the distance between pixel $(x,y)$ and the corresponding layer to the distance between the layer $j$ and $j-1$ at $(x,y)$ in relative amplitude space:

Apparently, $\sum_{i}W_{i}(x,y)=1$ and at location $(x,y)$ only features from layer $j$ and layer $j-1$ are integrated into the final FFMs. In [34], $\alpha$ is tuned to match human perception via physiological experiments. Here we learn the parameters of FFMs, $\alpha$ and $\sigma$, together with the policy from human gaze data directly.

Using FFMs as our state representation, we train a policy that mimics human gaze behavior using the IRL framework [43]. However, we found that the GAIL [18] IRL algorithm used in [43] is too sensitive to its hyper-parameters, due to its adversarial learning design, which is also shown in [23]. We therefore use IQ-Learn [12] as our IRL algorithm instead. Based on soft Q-Learning [14], IQ-Learn encodes both the reward and the policy in a single Q-function, and thus is able to optimize both reward and policy simultaneously.

Let $Q(s,a)$ be the Q-function, which maps a state-action pair $(s,a)$ to a scalar value representing the amount of future reward gained by taking action $a$ under state $s$. We want to find a reward function that maximizes the expected amount of cumulative rewards that the expert policy obtains over all other possible policies. Hence, IQ-Learn trains the Q-function by minimizing the following loss:

where $V(s)=\log\sum_{a}\exp(Q(s,a))$, $\rho_{E}$ and $\mathcal{P}$ denote the occupancy measure of the expert policy [18] and the dynamics, respectively. We do not apply the $\chi^{2}$-divergence proposed in [12] on the reward function since it did not lead to any notable improvement on our task. Given the learned Q-function $Q$, we can compute the reward as a function of the state and action:

and the policy as a function of the state:

$\tau$ is the temperature coefficient, controlling the entropy of the action distribution.

To this end, we train a binary classifier on top of the Q-function (see Sec. 3.2) for termination prediction using binary cross entropy loss. We weigh the loss computed on the termination and non-termination actions inversely proportionally to their frequencies. In addition, psychology studies [10, 41] have suggested that time could be an important ingredient in predicting stopping. However, we do not predict the duration of fixations in our model. Instead, we use the number of previous fixations as an approximation of time and concatenate it with the Q-values from the Q-function as input to train the termination classifier.

The sequence score (SS) has often been used to quantify the success of scanpath prediction [4, 43]. The sequence score is computed by an existing string matching algorithm that compares the two fixation sequences [33] after transforming them into strings of fixation cluster IDs. The fixation clusters are computed based on the fixation locations. However, we argue that the sequence score does not capture the semantic meaning of fixations which plays an important role in analyzing goal-directed attention: it only captures “where” a person is looking at but not “what” is being looked at. To this end, we propose the Semantic Sequence Score (SemSS), which transforms a fixation sequence into an object category sequence by leveraging the segmentation annotation provided in COCO [30]. Then, we apply the same string matching algorithm used in the traditional sequence score to measure the similarity between two scanpaths. Using the “things” versus “stuff” paradigm [6], we do not distinguish between object instances. In this paper, we focus on “thing” categories only, as we are interested in how non-target objects collectively affect human gaze behavior in visual search tasks. “Stuff” categories can be easily integrated into the semantic sequence score.

We compare our model with the following baselines: 1) human consistency, an oracle method where one searcher’s scanpath is used to predict another searcher’s scanpath; 2) detector, a ConvNet trained on target-present images of COCO-Search18 to output a target detection confidence map, from which we sample fixations sequentially with inhibition of return (IOR); 3) fixation heuristic, similar to detector, but trained to predict human fixation density maps using target-absent data; and more recent approaches including 4) IRL [43], and 5) Chen et al.’s model [7]. Note that Chen et al.’s model used a finer action space $30\times 40$. For fair comparison, we rescale its predicted fixations to our action space $20\times 32$.

As can be seen from Tab. 1, our method outperforms all other methods across all metrics in target-absent scanpath prediction²²2Both cIG and cNSS can only be computed for auto-regressive probabilistic models (our method, IRL, detector and fixation heuristic).. Our method is the closest to human consistency which is regarded as the ceiling of any predictive model. In the sequence score case, our method is only inferior to human consistency by 0.09, leading the second best (Chen et al. [7]) by 0.41. Excluding the effect of the termination predictor, the sequence scores of the first 2 and 4 fixations also show that even without terminating the scanpaths our method is still the best compared to all other computational models. Moreover, comparing the sequence scores of truncated scanpaths and full scanpaths, we see a trend of decreasing performance as the scanpath length increases for all methods, i.e., SS(2) $>$ SS(4) $>$ SS, and this pattern is particularly pronounced in target-absent search (there is no significant difference between SS and SS(4) for target-present search, see Tab. 3). The fact that later fixations during target-absent search are harder to predict suggests that human eye-movements behave more randomly at the later stage of search especially when there is no target in the scene.

We also qualitatively compare different methods by visualizing their predicted scanpaths for four scenes in Fig. 2. When searching for a microwave in this scene, our method alone predicted fixations on all three table and countertop surfaces in the image where microwaves are often found (similar to how a representative human searched). Similar phenomena are observed for the sink and knife searches. This shows that our method is able to capture the contextual relations between objects that play a role in driving target-absent fixations. When searching for the stop sign, our method was the only one that looked at the top of the centrally-located vertical object, despite heavy occlusion, speculatively because stop signs are usually mounted to the tops of poles. In contrast, IRL, which extracts features from blurred pixels using a pretrained ConvNet, completely failed to capture the vertical objects in this image that seem to be guiding search. This argues for the value in using our proposed FFMs to capture guiding contextual information extracted from peripheral vision.

In target-present search, human scanpaths are very consistent due to the strong guidance provided by the target object in the image. Indeed, a model trained with fixations from a group of people generalized well for a new unseen person [43]. However, given that there are large individual differences in termination time for target-absent search [8], we expect that individualized modeling may be necessary for target-absent search prediction. To test this hypothesis, we compared the predictive performance of group versus individual modeling of target-absent search fixations. The group model was trained with 9 subjects’ training scanpaths and tested on the testing scanpaths of the remaining subject. The individual model was trained with the training scanpaths of a single subject and tested on the same subject’s testing scanpaths. We did this for all 10 subjects.

Fig. 3 shows the comparison between the group model and the individual model in the sequence score of full scanpaths and truncated scanpaths (first four fixations) and the MAE of the length of the predicted scanpath. Interestingly, despite being trained with less data, the individual model shows better performance than the group model in full scanpath modeling, contrary to the group model being better in the truncated scanpath prediction. A critical difference between the modeling of truncated versus full scanpath is that the latter involves the search termination prediction. The rightmost graph in Fig. 3 also shows that the individual model generates less error (in MAE metric) in scanpath length prediction than the group model. These results altogether suggest that individualized modeling may be more suitable for target-absent search prediction. More experimental results on the termination criterion across different subjects can be found in supplementary.

First, we ablate the loss (see Eq. (8)) of our model by removing the auxiliary detection loss. Second, we ablate our proposed foveated feature maps (FFMs) by comparing it with dynamic contextual beliefs (DCBs) [43] and cumulative foveated image (CFI) [44] using the same IRL algorithm (i.e., IQ-Learn). As a finer-grained ablation, we ablate FFMs by using the features extracted by the FPN backbone of a COCO-pretrained Mask R-CNN as the state representation. We use the highest-resolution feature maps of FPN $P_{2}$. We further binarize the FFMs of our model such that the values are one at the fixated locations of the finest level (fovea) and the non-fixated locations of the coarsest level (periphery) and zero elsewhere. As shown in Tab. 2, the proposed auxiliary detection loss improves the performance in 5 out of 6 metrics. The semantic sequence score is increased from 0.476 to 0.516, which indicates that knowing the locations of non-target objects in the image is helpful for predicting the target-absent fixations. Comparing different state representations (i.e., FFMs, DCBs, CFI, FPN and binary masks), we can see that the proposed FFMs are superior to all other state presentations in predicting target-absent fixations. This shows the superiority of FFMs in representing the knowledge a human acquires through fixations compared to DCBs and CFI, which apply pretrained ConvNets on blurred images to simulate the foveated retina.

Despite being motivated by target-absent search, our method is also directly applicable to target-present fixation prediction. In this section, we compare our model with two competitive models, IRL [43] and Chen et al. [7], in target-present scanpath prediction. For fair comparison, we follow [43] and set the maximum scanpath length to be 6 (excluding the first fixation) for all models and automatically terminate the scanpath once the fixation falls in the bounding box of the target. Tab. 3 shows that our method achieves the best performance in 4 out of 6 metrics. Chen et al.’s model is slightly better than ours in semantic sequence score. They used a pretrained CenterNet [48] trained on COCO images [30] (about 118K images) to predict the bounding box of the target as input for their model, whereas we only used the target-present images in COCO-Search18 [8] (about 3K images) to train our object detection module (see Fig. 1). Despite being trained with less data, our model still outperforms Chen et al. [7] in the other five metrics, especially when evaluated in truncated fixed-length scanpaths (i.e., SS(2) and SS(4)). We further expect our model to perform better when using all COCO training images to train our object detection module. Tab. 1 and Tab. 3 together demonstrate that our proposed method not only excels in predicting target-absent fixations (see Sec. 4.3), but also target-present fixations.