Constructing a Visual Relationship Authenticity Dataset

The Visual Phrases dataset [Sadeghi and Farhadi, 2011] contains 17 phrases and 8 objects on the Pascal VOC2008 dataset [Everingham et al., 2010]. The Scene Graph dataset consists of not only visual relationships but also the attributes of objects [Johnson et al., 2015]. Their dataset is mainly constructed for image retrieval, which contains 109,535 instances of relationships in 5,000 images. The VRD dataset contains 100 object categories and 70 predicates for 5,000 images [Lu et al., 2016]. VG is a large-scale dataset containing more than 100k images, annotated with various labels such as objects, attributes, relations, and scene graphs [Krishna et al., 2016]. Being different from these existing datasets, our dataset is the only one that annotates the authenticity of visual relationships. A statistical comparison of our dataset and previous ones are shown in Table 1.

The most challenging problem in visual relationship detection is the data sparseness of relationships, and many studies have been conducted to address this problem. ?) proposed a model that handles objects and predicates independently. They then combined them for visual relationship detection. ?) used linguistic knowledge from both the dataset and Wikipedia to regularize the visual model. ?) used various visual features, such as the appearance, size, position, attribute of objects, and spatial relationships between objects. ?) treated the subject and object in a visual relationship as context, and the predicate as interaction. They modeled the interaction with context in multiple relationships for visual relationship detection. ?) pointed out the importance of language bias and spatial features, and fused them for visual relationship detection. ?) proposed a weakly-supervised model learnt from image-level labels. ?) simultaneously selected object pairs and classified them for both object and relationship detection. ?) first learnt a prior by a multi-relational learning model, and then used a factorization scheme for the prior. ?) used a recurrent neural network to model the semantic connection among objects. ?) transferred visual phrase embeddings from triplets in the training data to unseen test triplets using analogies between relationships containing similar objects. ?) used unlabeled relationships to improve the accuracy of visual relationship detection.

Our dataset is based on the Flicker30k entities dataset [Plummer et al., 2015]. Flickr30 entities is a large-scale dataset with images and captions, in which the entities in the captions are localized to the regions in the images. Following [Plummer et al., 2015], we use 29,769, 1,000, and 1,000 samples as training, validation, and testing splits, respectively. An example of an image and its captions from the Flickr30k entities dataset is shown in Figure 1. Each image has 5 captions, and the entities in the captions are localized by their corresponding regions in the image. In this example, all entities in the captions have corresponding regions in the image; however, there can be more abstract entities related to events and scenes, such as “break” and “a minute” that do not have corresponding visual concepts and thus have no corresponding regions in the image.

In the Flickr30k entities dataset, entities in captions are annotated by chunking noun phrases in the Flickr30k captions [Young et al., 2014]. These entities are further categorized into types such as people, body parts, and clothing etc. using a manually constructed dictionary [Plummer et al., 2015]. As shown in Figure 3, visual relationship candidates are generated by combining two entities in a caption. Specifically, all pairs of entities in a single caption are combined as candidate relationships, where the subjects and objects must be the ones in which they appear in the caption. We define the phrase lying between two entities as predicate. More specifically, the phrase that comes right before the second entity in the caption is treated as the predicate. In Figure 3, three visual relationship candidates are generated.

Some phrases lying between two entities like “,”, “, and”, “while” and “space” are not predicates and thus are removed from the group of candidates. Entities in the dataset have type tags describing their characteristics (such as “people” or “clothing”). The entities having the type tag “notvisual” do not qualify as candidates for visual relationships, because they are not visual in the image. The statistics of the generated visual relationship candidates are shown in the first row of Table 2.

If a caption correctly represents image content and has correct dependency relations, we can detect true visual relationships based on the dependency relationship between two entities. In this section, we describe the method for detecting true visual relationships via dependency parsing. Dependency parsing of the captions is done by the Stanford parser.³³3https://nlp.stanford.edu/software/lex-parser.shtml The parsed captions are cross checked with the candidates for visual relationships obtained in Section 3.2. and then true visual relationships are extracted.

In detail, the detection of true visual relationships are carried out in the following manner.

1. 1.

   Do dependency parsing on the captions and construct a directional graph, of which nodes are words.⁴⁴4Collapsed-dependencies are treated as a directional graph.

2. 2.

   Merge nodes in the graph based on the entities and predicates in Flickr30k. The edges connecting the same nodes are ignored.

3. 3.

   Get the visual relationship if the entities and their predicate belong to predefined types.

4. 4.

   Extract the visual relationships obtained in Step 3 as true visual relationship if they are also contained in the candidates generated in Section 3.2.

Steps 3 and 4 are named as type extraction. The “predefined types” in Step 3 refer to the dependency relationship of the entity pair and their predicate. As dependency parsing is not error-free, we empirically decide the combination of types. Let EN1 (i.e., subject) and EN2 (i.e., object) be the pair of entities and RE be the predicate between them. As long as EN1, EN2, and RE are linked in the graph, the combinations that make a valid visual relationship are the following 7 types.

• •

  A: RE $\rightarrow$ EN1, RE $\rightarrow$ EN2

• •

  B: EN1 $\rightarrow$ RE, EN1 $\rightarrow$ EN2

• •

  C: EN1 $\rightarrow$ PRE, EN2 $\rightarrow$ RE

• •

  D: RE $\rightarrow$ EN1, EN2 $\rightarrow$ EN1

• •

  E: EN1 $\rightarrow$ RE $\rightarrow$ EN2

• •

  F: RE $\rightarrow$ EN1 $\rightarrow$ EN2

• •

  G: EN1 $\rightarrow$ EN2 $\rightarrow$ RE

A and B are cases of having a common parent node, and C and D are cases of having a common child node.

Visual relationships that could not be extracted via dependency parsing are manually annotated via a crowdsourcing service, Amazon Mechanical Turk (AMT). The screenshots of the instruction part and some annotation tasks in AMT are shown in Figures 5 and 6, respectively. Our interface covers a single session for multiple annotation tasks. The first part of our interface provides the instructions for annotation, some annotation examples with correct and incorrect annotation results, and their reasons. It also notifies that there are dummy questions and workers who fail to answer these dummy questions would be rejected. Our dummy questions are the same annotation task but are preliminarily annotated so that we can automatically evaluate the dummy questions.

Below the instruction part, the interface continues to actual annotation tasks. It shows 10 panels (on average) like Figure 6, each of which covers a single image (and thus have multiple annotation tasks for the image). Each image yields 1 to 10 candidate visual relationships to be annotated. We put many panels as a single session, which contains 50 annotation tasks. An annotation task is simply choosing Yes/No on a radio button. A single session comes with 5 dummy questions. A worker who achieves an accuracy higher than 0.8 on the dummy questions is accepted and otherwise are rejected. Figure 6 shows an example, where the second question about the visual relationship “man with a fish” is a dummy question. The reward for a single session was set to $0.2, and we only recruited workers who have the AMT master qualification.

Each session serves as a human intelligence task (HIT) in AMT. The overall flow of our AMT-based annotation is as follows.

1. 1.

   Issue HITs in AMT.

2. 2.

   Workers annotate HITs.

3. 3.

   The accuracy of a single HIT is computed as the accuracy over the dummy questions.

4. 4.

   If the accuracy is higher than 0.8, we accept the HIT. Otherwise we reject it.⁵⁵5Only workers whose HIT is accepted get the reward. Rejected HITs yield no reward and are not included in the dataset.

5. 5.

   The responses from the workers are aggregated and only candidate visual relationships for which the majority (i.e., more than 3 out of 5) of the responses are Yes are labelled as true; otherwise, they are labelled as false.