The Dialog Must Go On:
Improving Visual Dialog via Generative Self-Training

Self-training. We have a labeled dataset $L$ = $\{(x_{n},y_{n})\}_{n=1}^{N}$ and an unlabeled dataset $U$ = $\{\tilde{x}_{m}\}_{m=1}^{M}$. Typically, self-training trains a teacher model $P_{\mathcal{T}}$ on the labeled dataset $L$. The teacher then predicts the pseudo label $\tilde{y}$ for the unlabeled data $\tilde{x}\sim U$, constructing the pseudo-labeled dataset $\tilde{L}$ = $\{(\tilde{x}_{m},\tilde{y}_{m})\}_{m=1}^{M}$. Finally, a student model $P_{\mathcal{S}}$ is trained on $L\cup\tilde{L}$. Many variants have been studied on this setup: (1) selecting the subset of the pseudo-labeled dataset [23, 92, 84], (2) adding noise to inputs [98, 23, 92, 91, 84], and (3) iterating the above setup multiple times [23, 92].

Visual dialog. The visual dialog (VisDial) dataset [12] contains an image $v$ and a visually-grounded dialog $d$ = $\{\underbrace{c}_{d_{0}},\underbrace{(q_{1},a^{gt}_{1})}_{d_{1}},\cdots,\underbrace{(q_{T},a^{gt}_{T})}_{d_{T}}\}$ where $c$ denotes an image caption. $T$ is the number of rounds for each dialog. At round $t$, a dialog agent is given a triplet $(v,d_{<t},q_{t})$ as an input, consisting of the image, the dialog history, and a visual question. $d_{<t}$ denotes all dialog rounds before the $t$-th round. The agent is then expected to predict a ground-truth answer $a^{gt}_{t}$. There are two broad classes of methods in VisDial: generative and discriminative. Generative models aim to generate the ground-truth answer by maximizing the log-likelihood of $a^{gt}_{t}$. In contrast, discriminative models are trained to retrieve the ground-truth answer from a list of answer candidates $a^{gt}_{t}$ $\in$ $\left\{a_{t}^{1},\cdots,a_{t}^{100}\right\}$. Our main focus is the generative models since they do not need pre-defined answer candidates and are thus more practical to be deployed in real-world applications.

Teacher & questioner training. First, a series of question-and-answer pairs for the unlabeled images should be generated to train the answering agent. Accordingly, GST first trains the answer generator, the teacher model $P_{\mathcal{T}}$, on the gold VisDial dataset. Specifically, the teacher learns to generate the ground-truth answer’s word sequence $a_{t}=(w_{t,1},\cdots,w_{t,S})$, given the context triplet $c_{t}\triangleq(v,d_{<t},q_{t})$, consisting of the image, the dialog history, and the question. It is optimized by minimizing the negative log-likelihood of the ground-truth answer. Formally,

$$\begin{split}\mathcal{L}_{\mathcal{T}}&=-{1\over NT}\sum_{n=1}^{N}\sum_{t=1}^{T}\mathrm{log}\,P_{\mathcal{T}}(a_{n,t}|c_{n,t})\\ &=-{1\over NTS}\sum_{n=1}^{N}\sum_{t=1}^{T}\sum_{s=1}^{S}\mathrm{log}\,P_{\mathcal{T}}(w_{s}|c_{n,t},w_{<s})\end{split}$$

(1)

where $N$, $T$, and $S$ denote the number of data tuples in gold VisDial data, dialog rounds, and the sequence length of the ground-truth answer, respectively. $w_{<s}$ indicates all word tokens before the $s$-th token in the answer sequence. Similar to the teacher, the questioner is trained to generate the question at round $t$, given the image and the dialog history until round $t-1$ (i.e., $P_{\mathcal{Q}}(q_{t}|v,d_{<t})$). The questioner is also optimized by minimizing the negative log-likelihood of the follow-up question. Note that the teacher and the questioner are trained separately to prevent possible unintended co-adaptation [37]. Both the teacher and the questioner are based on encoder-decoder architecture, where an encoder aggregates the context triplet, and a decoder generates the target sentence. We implement the models by integrating a pre-trained vision-and-language encoder, ViLBERT [54], with the transformer decoder [72]. We refer readers to Appendix A for a detailed architecture.

Unlabeled in-domain image retrieval (IIR). Inspired by the work [16] that highlighted the importance of using in-domain data, GST retrieves in-domain image data from the Conceptual 12M dataset [7] with an out-of-distribution (OOD) detection model. Specifically, we extract the $D$ dimensional feature vector for each image in the gold VisDial dataset by using the Vision Transformer (ViT) [15] in the CLIP model [67], yielding a feature matrix for the entire images $\mathbf{X}=(X_{1},\cdots,X_{N})^{\top}\in\mathbb{R}^{N\times D}$. Based on the matrix, we build the multivariate normal distribution whose dimension is $D$, i.e., $\mathbf{X}\sim\mathcal{N}_{D}(\mu,\Sigma)$. We regard this normal distribution as the empirical distribution of the gold VisDial images and perform OOD detection by identifying the probability of each feature vector for the unlabeled image. Consequently, the top-$M$ unlabeled images are retrieved out of 12 million Web images ($M\approx 3.6$ million).

Visually-grounded dialog generation. This step mimics a scenario where two people have a conversation about the given images. Given the retrieved images $U=\{\tilde{v}_{m}\}_{m=1}^{M}$, our goal is to generate the visually-grounded dialogs $\{\tilde{d}_{m}\}_{m=1}^{M}$ where each dialog $\tilde{d}$ consists of the image caption and $T$ rounds of QA pairs. In an actual implementation, we use the image captions in the Conceptual 12M dataset [7] and thus do not generate the captions. The QA pairs are sequentially generated. Concretely, the image $\tilde{v}$, the caption $\tilde{c}$, and the generated QA pairs until round $t-1$ are used as inputs when the questioner generates the question at round $t$ (i.e., $\tilde{q}_{t}$). After then, the teacher produces the corresponding answer $\tilde{a}_{t}$ based on the image $\tilde{v}$, the dialog history $\tilde{d}_{<t}$, and the question $\tilde{q}_{t}$. Finally, GST produces the silver VisDial dataset $\tilde{L}=\{(\tilde{v}_{m},\tilde{d}_{m})\}_{m=1}^{M}$.

Student training with noisy data. In Figure 1, the student $P_{\mathcal{S}}$ is trained on the combination of the silver and the gold VisDial data. According to many studies [92, 23, 84, 98] in self-training, selectively utilizing the samples in the pseudo-labeled dataset is a common approach since the confidence of the teacher model’s predictions varies from sample to sample. To this end, we introduce a simple yet effective data selection method for the sequence generation problem, perplexity-based data selection (PPL). PPL is to utilize the answers whose perplexity of the teacher is below a certain threshold. Perplexity is defined as the exponentiated average negative log-likelihood of a sequence; the lower, the better. We hypothesize that PPL, albeit noisy, can be an indicator of whether the generated answer is correct or not, as in [81]. Furthermore, inspired by the consistency regularization [91, 84] widely utilized in recent SSL algorithms, we also propose the multimodal consistency regularization (MCR) to improve the generalization capability of the student. MCR encourages the student to yield predictions similar to the teacher’s predictions even when the student is provided with perturbed multimodal inputs. Finally, we design a loss function for the student as:

$$\small\begin{split}&\mathcal{L}_{\mathcal{S}}=\\ &~{}-{1\over MT}\sum_{m=1}^{M}\sum_{t=1}^{T}\mathbbm{1}(\mathrm{PPL}(\tilde{a}_{m,t})<\tau)\mathrm{log}\underbrace{P_{\mathcal{S}}(\tilde{a}_{m,t}|\mathcal{M}(\tilde{c}_{m,t}))}_{\mathrm{MCR}}\\ &~{}-{1\over NT}\sum_{n=1}^{N}\sum_{t=1}^{T}\mathrm{log}P_{\mathcal{S}}(a_{n,t}|c_{n,t})\\ &~{}\mathrm{where}\,\,\,\mathrm{PPL}(\tilde{a}_{t})=\mathrm{exp}\left\{{-{1\over S}}\sum_{s=1}^{S}\mathrm{log}P_{\mathcal{T}}(\tilde{w}_{s}|\tilde{c}_{t},\tilde{w}_{<s})\right\}\end{split}$$

(2)

where $M$, $\mathbbm{1}$, and $\tau$ denote the number of data tuples in silver VisDial data, indicator function, and selection threshold, respectively. $\tilde{c}_{m,t}\triangleq(\tilde{v}_{m},\tilde{d}_{m,<t},\tilde{q}_{m,t})$ denotes the context for the silver VisDial data. The loss function is the sum of the losses for the silver and the gold VisDial data. PPL and MCR are applied to compute the loss of the silver VisDial data. PPL is used in the indicator function above, selecting the synthetic answers whose perplexity of the teacher is below the threshold $\tau$. It implies that the unselected answers are ignored during training. The teacher’s perplexity of each answer is computed in the dialog generation step above. Next, $\mathcal{M}$ denotes the stochastic function for MCR that injects perturbations to the input space of the student. Inspired by ViLBERT [54], we implement the stochastic function by randomly masking 15% of image regions and word tokens. Specifically, masked image regions have their image features zeroed out, and the masked word tokens are replaced with a special [MASK] token. The intuition behind MCR is minimizing the distance between the perturbed (i.e., masked) predictions from the student and the unperturbed predictions (i.e., $\tilde{a}_{m,t}$) from the teacher. It indicates that the perturbation is not injected when the teacher generates the synthetic answers. We believe MCR makes the student robust to the input noise, and PPL encourages the student to maintain a low entropy (i.e., confident) in noisy data training. The student and the teacher have the same model capacity and are based on the same model architecture.

VisDial datasets. We evaluate our proposed approach on the VisDial v1.0 and v0.9 datasets [12], collected by the AMT chatting between two workers about MS-COCO [52] images. Each dialog consists of a caption from COCO and a sequence of ten QA pairs. The VisDial v0.9 dataset has 83k dialogs on COCO-train and 40k dialogs on COCO-validation images. More recently, Das et al. [12] released additional 10k dialogs on Flickr images to use them as validation and test splits for the VisDial v1.0 dataset. As a result, the VisDial v1.0 dataset contains 123k, 2k, and 8k dialogs as train, validation, and test split. This dataset is licensed under a Creative Commons Attribution 4.0 International License.

Evaluation protocol. We follow the standard evaluation protocol established in the work [12] for evaluating visual dialog models. The visual dialog models for both generative and discriminative tasks have been evaluated by the retrieval-based evaluation metrics: mean reciprocal rank (MRR), recall@k (R@k), mean rank (Mean), and normalized discounted cumulative gain (NDCG). Specifically, all dialogs in VisDial contain a list of 100 answer candidates for each visual question, and there is one ground-truth answer in the answer candidates. The model sorts the answer candidates by the log-likelihood scores and then is evaluated by the four different metrics. MRR, R@k, and Mean consider the rank of the single ground-truth answer, while NDCG¹¹1https://visualdialog.org/challenge/2019#evaluation considers all relevant answers from the 100-answers list by using the densely annotated relevance scores for all answer candidates. The community regards NDCG as the primary evaluation metric.

The size of synthetic data. The size of the silver VisDial data (i.e., $M$) is 3.6M which is 30x larger than that of the gold VisDial data ($N=0.12$M). Note that the silver VisDial data contains approximately 36M QA pairs since each dialog contains 10 QA pairs. 11.7M QA pairs out of 36M ($\sim$32%) are actually utilized after applying perplexity-based data selection when we set the selection threshold $\tau$ to 50. Consequently, the total amount of the training data is nearly 12.9M QA pairs, combining the silver data (11.7M QA pairs) with the original gold data (1.2M QA pairs).

Iterative training. We introduce the concept of iterative training [92, 23], which iterates the self-training algorithm a few times. The iterative training treats the student model at $i$-th iteration as a teacher model at ($i$+1)-th iteration to generate a new synthetic silver data and train a new student. Specifically, the iterative training repeats the third and fourth steps in Figure 1, where the silver VisDial data accumulates as the iteration proceeds. The student model at each iteration is trained with the accumulated silver and gold data by following the previous studies [92, 23]. We iterate GST up to three times. Unless stated otherwise, the student model is trained with three iterations.

Comparison with state-of-the-art. We compare GST with the state-of-the-art approaches on the validation set of the VisDial v1.0 and v0.9 datasets, consisting of UTC [8], MITVG [9], VD-BERT [89], LTMI [60], KBGN [28], DAM [29], ReDAN [19], DMRM [10], Primary [22], RvA [61], CorefNMN [40], CoAtt [90], HCIAE [55], and MN [12]. We decided to use the validation splits since all previous studies benchmarked the models on those splits. In Table 1, GST significantly outperforms all compared methods on all evaluation metrics. Compared with the state-of-the-art model, the student model improves MRR 3.20% (56.83 $\rightarrow$ 60.03) and R@1 3.26% (47.14 $\rightarrow$ 50.40) on the VisDial v0.9 dataset. The improvement is consistently observed on the VisDial v1.0 dataset, boosting NDCG 1.61% (63.86 $\rightarrow$ 65.47) and MRR 0.97% (52.22 $\rightarrow$ 53.19). Moreover, it is noticeable that recent strong models (i.e., UTC, MITVG, and VD-BERT) are also built based on the pre-trained weights of ViLBERT [54], transformer [87], and BERT [14], respectively. Our proposed method also achieves new state-of-the-art results on the discriminative VisDial models. Details can be found in Appendix B.

GST in the low-data regime. Is GST also helpful when gold data is scarce? We investigate this question to identify the effect of GST in the low-data regime. We assume that only a small subset of the gold VisDial data (1%, 5%, 10%, 20%, and 30%) is available. Therefore, the size of the gold data is 0.01$N$, 0.05$N$, 0.1$N$, 0.2$N$, and 0.3$N$, respectively. We first train the teacher and the questioner on such scarce data, and then these two agents generate a new silver VisDial data for unlabeled images in the Conceptual 12M dataset [7] with size $5N$. The student is then trained on the newly generated silver VisDial data and the small amount of the gold VisDial data. The student is based on a single iterative training, and PPL and MCR are still applied in this experiment. In Table 2, GST yields huge improvements on both metrics, especially NDCG, boosting up to 11.09 absolute points compared with the teacher. We observe that the smaller the amount of gold data, the larger the performance gap between the teacher and the student on NDCG. It implies that GST is helpful, especially when gold data is scarce. We speculate the results in the low-data regime are particularly remarkable in other dialog-based tasks [86, 2, 69, 51] since they are based on relatively small-scaled datasets, and scaling up the size of the human-dialog datasets is laborious and expensive.

Question type analysis. We conduct a question-type analysis to identify what type of questions obtain benefits from GST. We divided the question type into six categories, Yes/No, Color, Objects, Counting, Time/Place, and Others. In Table 3, the student model obtains more gains compared with the teacher model in less frequent question types (e.g., Counting and Time / Place).

Baselines. We compare our student model against two ablative baselines: (1) the teacher model and (2) the student model utilizing the entire CC12M images without applying the in-domain image retrieval (i.e., student-iter1-full). We propose the student-iter1-full model to study the effect of the discarded images and the corresponding synthetic dialog data on adversarial robustness.

Adversarial robustness against the FGSM attack. The Fast Gradient Signed Method (FGSM) [20] is a white-box attack that perturbs the visual inputs based on the gradients of the loss with respect to the visual inputs. Formally,

$$\begin{split}\mathrm{FGSM}(x)=x+\epsilon\cdot\mathrm{sign}(\nabla_{x}\mathcal{L}(x,y))\end{split}$$

(3)

where $x$ and $y$ denote the visual inputs and the corresponding ground-truth labels, respectively. $\epsilon$ is a hyperparameter that adjusts the intensity of perturbations. However, different from the above setup, each question in VisDial can have one or more relevant answers in the list of answer candidates. We thus define the FGSM attack for VisDial as follows:

$$\begin{split}\mathrm{FGSM}(v)=v+\epsilon\cdot\mathrm{sign}(\sum_{c=1}^{C}r(a_{t,c})\cdot\nabla_{v}\mathcal{L}(c_{t},a_{t,c}))\end{split}$$

(4)

where $C=100$ and $r(\cdot)$ denote the number of answer candidates and a function that returns the human-annotated relevance scores for each answer candidate, respectively. The relevance scores range from 0 to 1. $c_{t}$ and $a_{t,c}$ are the context triplet (i.e., $c_{t}\triangleq(v,d_{<t},q_{t})$) and the $c$-th answer candidate, respectively. The Equation 4 indicates that the gradients of the loss for all relevant answers are considered for the FGSM attack.

Adversarial robustness against the textual attacks. We also study the adversarial robustness against textual attacks to illustrate the effect of GST. We decide to perturb the dialog history because it contains useful information to answer the given question (e.g., cues for pronoun). However, according to recent studies [1, 34] in VisDial, not all questions require the dialog history to respond with the correct answers. So the work [1] has proposed a challenging subset of the VisDial validation dataset called VisDialConv. The VisDialConv dataset only contains questions that necessarily require the dialog history to answer (e.g., can you tell what it is for?). The crowd-workers conducted a manual inspection to select such context-dependent questions.

Comparison between silver and gold data. For qualitative analysis of the silver data, we visualize the generated conversations from our proposed models and the ones from humans. We excerpt the human conversation from the VisDial v1.0 validation dataset, and the questioner and the student generate the machine conversation using the image and the caption in the validation data. As shown in Figure 3, diverse visual questions are generated in the silver VisDial data. For example, in D10 of the last example, the questioner asks about “a car” not mentioned by the human questioner and not even presented in the image caption. The student responds correctly to the question. Likewise, from D3 to D6 in the first example, the questioner deals with “a cell phone,” whereas the human questioner deals with different topics. However, we identify that the student sometimes fails to generate correct answers (i.e., the red-colored text), showing the importance of more precise visual grounding.

The diversity of silver questions. We further quantify the generated question’s diversity by comparing the gold questions with the silver ones for the same images in the VisDial v1.0 validation dataset. We extract N-grams for every ten questions (i.e., per image) in the gold and silver data and compare the N-grams between the two. We define the question diversity as the percentage of unique silver N-grams not observed in the gold N-grams. We identify the question diversity by adjusting N from one to four. We generate three silver datasets and report the mean and standard deviations of the question diversity since the questioner performs stochastic decoding (see Appendix D). In Table 5, the diversity significantly increases as N increases (92.80% at N=4). It indicates that the questioner mainly generates different and distinctive 4-grams compared with the human questioner.