Understanding Spatial Relations through Multiple Modalities

This task is to predict the spatial relation $R$ given the subject $S$ and the object $O$. For the subject $S$, we have the corresponding text information $T_{s}$, position information $P_{s}$, and image information $I_{s}$. Similarly, we have the corresponding text information $T_{o}$, position information $P_{o}$, and image information $I_{o}$ for the object $O$.

Feed Forward Network (FF). For the text information, we use average (for multi-word subjects and objects) glove embeddings [Pennington et al., 2014] of the words in the text to represent it. For simplicity, we use $w_{s}$ and $w_{o}$ to represent the average glove embbeddings for the $T_{s}$ and $T_{o}$. As for the position $P$, it contains $4$ float values denoting the $x$ and $y$ coordinate of the subject(object) center and the half-width and half-height of the bounding box of the subject(object). We then pass $w_{s}$, $w_{o}$, $P_{s}$, $P_{o}$ through a feed forward network with $128$ neurons and take a softmax of the predictions of all possible relations:

$$h=[\sigma(W_{s}[w_{s};P_{s}]+b_{s});\sigma(W_{o}[w_{o};P_{o}]+b_{o})]$$

$$\hat{p}=softmax(W_{h}h+b_{h})$$

where $\sigma$ is the activation function ReLU.
Feed Forward Network + Image Embeddings (FF+I). In addition to the text information and position information of the subject and object, we use pre-trained visual embeddings to represent the image information $I_{s}$ and $I_{o}$. For simplicity, we use $i_{s}$ and $i_{o}$ to represent the visual embeddings for the subject and the object. The visual embeddings of the words are provided by  [Collell and Moens, 2018] from a VGG128 network [Chatfield et al., 2014] pre-trained on Imagenet and fine-tuned on Visual Genome  [Krishna et al., 2016]. The hidden representation in the FF + Image Embeddings are as following:

$$h=[\sigma(W_{s}[w_{s};P_{s};i_{s}]+b_{s});\sigma(W_{o}[w_{o};P_{o};i_{o}]+b_{o})]$$

BERT. We use BERT [Devlin et al., 2018] as language model to predict the most likely spatial relation by masking the relation and providing ”subject [MASK] object” as input and running the beam search to obtain top $20$ predictions.
Fine-tuned BERT (f-BERT). We fine-tune BERT for the Visual Genome datatset by collecting all the ”subject relation object” texts from the training data.

We combine the prediction of our best model (FF+I) with normal BERT and fine-tuned BERT to get two combined models. We essentially re-rank the predictions from the two models. Assume $\hat{p}_{bert}$ and $\hat{p}_{ff}$ are the predicted probability distribution from the BERT and FF+I models, the predicted probability of Spatial BERT is: $\hat{p}=\hat{p}_{ff}+\lambda\hat{p}_{bert}$, where $\lambda$ is a non-negative float to adjust the weight of the BERT predictions.

We use the Visual Genome dataset  [Krishna et al., 2016]. We work on the (subject, relation, object) triples where the subject and object is accompanied by the bounding box information²²2Note each image is scaled to $1$ so that the sizes are comparable. See  [Collell et al., 2018] for more information on the data-preprocessing step.. The dataset is partitioned into two categories: explicit and implicit based on the spatial relation in a triple and the experiments are performed separately for each category. There are $333321$ implicit and $682374$ explicit triples. In Figure 2 we see examples from the dataset for the $explicit$ relations between $cat$ as $subject$ and $chair$ as $object$. In Figure 3 we see examples from the dataset for the $implicit$ relations between $man$ as $subject$ and $surfboard$ as $object$.

For the explicit relations, on is the majority relation with frequency $425308$ and the majority baseline for explicit relation prediction is ${\bf 62}$%. Similarly, for the implicit relations, has is the majority relation with a frequency $167780$ and the majority baseline for implicit relation prediction is ${\bf 50}$%.

We see that the best performance in several of the settings is achieved by Spatial BERT. Although the gains may look small, the number of data-points is huge and thus, the improvement is significant in terms of absolute counts. For smaller percentages of training data, the spatial models perform poorly but later on beats BERT. Also, notice BERT performs significantly better for the explicit spatial relations than the implicit spatial relations possibly because the set of implicit relations is much larger and the task of predicting them requires more image understanding and common-sense compared to the explicit spatial relations. BERT and f-BERT performances do not change across different data percentages because they are both just used for inference and the amount of training data does not affect them.

If a spatial relation was seen during training – (man, riding, horse) and the supporting image, the model should be able to infer that the implicit (in this example) spatial relation for a new image depicting (lady, riding, elephant) should be riding, even if it has never seen the subject (lady) or object (elephant) before. We show two illustrative examples from Visual Genome that we want to handle for the unseen subject (Figure 5) and unseen object (Figure 6) settings respectively. The pre-trained embeddings of the unseen subject(object) is similar to the embeddings of similar seen subject(object) and this (along with the positional information) should help the model identify that the relations are similar. To systematically test this capability, we perform experiments for the implicit and explicit relations in Table 2. For each dataset type we experiment with three settings – unseen subject, unseen object and unseen relation.

The unseen subject(object) setting is relatively easier compared to the unseen relation setting. For subject, we test if we can correctly predict the flying relation in (kid, flying, kite) even if we have never seen (kid, flying, $X$) during training. For objects, we test if we can correctly predict the riding relation in (man, riding, elephant) even if we have never seen ($X$, riding, elephant) during training. Here, $X$ denotes any object or subject, respectively. We first tabulate all the (subject, relation), (object,relation) pairs and we split this list into the test set pairs($15\%$) and the training and development set pairs($85\%$). We then form the test set (train, development) by collecting all data-points whose (subject,relation), (object,relation) is in the test( train or development) set pairs. Thus, the test set has (subject,relation), (object,relation) which are not seen during training. We only use the normal pre-trained BERT for the generalized experiments since fine-tuning is not very natural for these settings.

Unseen Subject. For the explicit relations, we see that Spatial BERT performs much better than either model in isolation. This is potentially because the explicit relations are a smaller set and easier for BERT to predict and this in turn helps Spatial BERT. However, since the class of implicit relations are much larger and subtle BERT performs very poorly and spatial model by itself performs the best for the implicit relations.
Unseen Object. As shown in Table 2, for both the explicit and the implicit relations Spatial BERT gives the best performance although BERT by itself does not perform very well.
Unseen Relation. This task is possible because BERT is able to predict a much larger class of relations and using our GloVe-scoring metric we can decompose the scores of these relations across the known set of relations and this should bring the correct relations towards the top of the ranked list. We also relax the accuracy metric by counting a prediction as correct if the gold relation is in the top-$5$ predicted relations. In this setting, the spatial model by itself cannot predict any unseen relation and thus, BERT gives the best performance.