MMIU: Dataset for Visual Intent Understanding in Multimodal Assistants

Multimodal assistants leverage vision as an additional input signal along with other modalities. However, the identification of user intent becomes a challenging task as the visual input might influence the outcome. Current digital assistants rely on spoken input and try to determine the user intent from conversational or device context. However, a dataset which includes visual input (i.e., images or videos) corresponding to questions targeted for multimodal assistant use cases is not readily available. While work in visual question answering (VQA) and visual question generation (VQG) is an important step forward, this research does not capture questions that a visually-abled person would ask multimodal assistants. Moreover, several questions do not seek information from external knowledge Jain et al. (2017); Mostafazadeh et al. (2016). Recently, the OK-VQA dataset Marino et al. (2019) tries to address this shortcoming by including questions that need to reason over unstructured knowledge. However, we make two main observations about its unsuitability for multimodal assistant use cases. Firstly, the image types in OK-VQA datasets are often not appropriate to allow meaningful questions to be posed to the digital assistant. Secondly, the OK-VQA dataset has many obvious or common-sense questions pertaining to its images, as shown in Fig. 1, which are not challenging enough to ask a digital assistant.

The task of identifying the intent in the given question could be challenging because of the ambiguity that can arise due to the visual context in the image. For example, as shown in the Fig. 2, the same question can have different intents based on the visual contents. Thus, the intent understanding process must take into account both the question and the image to correctly identify the intent. Various techniques have been proposed to combine textual and visual features to perform joint understanding. These approaches mainly use either fusion-based methods to combine the independently learned features for both the modalities and then use this joint representation for a given task, or use attention-based methods where a joint representation is learned by attending to relevant parts of the modalities simultaneously Nguyen and Okatani (2018); Tan and Bansal (2019); Lu et al. (2019); Chen et al. (2020). We provide comprehensive experiments of various image and text representation strategies and its effect on intent classification.

To address the dataset issue, we introduce an effective dataset of images and corresponding natural questions that are more suitable for a multimodal assistant. To the best of our knowledge, this dataset is the first of its kind. We call it, the MMIU (Multi-modal Intent Understanding) dataset. We collected about 12K images and asked annotators to come up with questions that they would ask a multimodal assistant. Thus, we obtained 44K questions for 12K images, ensuring their applicability to digital assistants. We then created an annotation task where given an (image, question) pair, the annotators provide the underlying intent. Based on the nature of data, we pre-determined 14 different intent classes for annotators to choose from. Our dataset includes questions for factoid/descriptive information, searching for local business, asking for the recipe of food items, navigating to a specific address, chit-chat conversation about visual contents, and translating observed foreign text into the target language.

We then build a multi-class classification model that leverages visual features from the image and textual features from the question to classify a given (image, question) pair into 1 of 14 intents. To understand the effect of visual features, we use pre-trained CNNs such as VGG19 Simonyan and Zisserman (2015), ResNet152 He et al. (2016), DenseNet161 Huang et al. (2017), Inceptionv3 Szegedy et al. (2014), MobileNetv2 Sandler et al. (2018) to get the image representation. We also experiment with recent vision transformers such as ViT Dosovitskiy et al. (2020) to see if they do any better at this task compared to traditional CNNs. To understand the role of textual features derived from the question, we use popular transformer-based text representation strategies such as BERT Devlin et al. (2018), RoBERTa Liu et al. (2019), ALBERT Lan et al. (2019), DistillBERT Sanh et al. (2019) to get the contextual representation of the question. We also experiment with combining these two modalities using early and late fusion approaches to see the overall effect on performance. Finally, we leverage a few state-of-the-art multimodal transformers such as ViLBERT Lu et al. (2019), VL-BERT Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei (2020), LXMERT Tan and Bansal (2019), UNITER Chen et al. (2019) which have shown impressive results on various vision and language tasks, to check their effect on our intent classification task.

We use standard evaluation metrics commonly used for multi-class classification Grandini et al. (2020). The results of some selected experiments are shown in Table 1. In our early results, we notice that the use of text-only features dominates the intent classification task. However, the best weighted-F1 score with DistillBERT is far from ideal. The results from the fusion approaches indicate that the vanilla fusion methods are not effectively leveraging the image modality during classification. Moreover, leveraging off-the-shelf multimodal transformers such as LXMERT does not seem to help