Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System

Dialogue systems are often categorized as either task-oriented or open-ended, where the task-oriented dialogue systems are designed to fulfill specific tasks and handle goal-oriented conversations, whereas the open-ended systems or chat-bots allow more generic conversations such as chit-chat [Jurafsky and Martin, 2018]. With the advancements of deep learning-based language technologies and increased availability of large datasets in the research community, the end-to-end trained dialogue systems are shown to produce promising results for both goal-oriented [Bordes et al., 2016] and open-ended [Dodge et al., 2015] applications. Dialogue Managers (DM) of goal-oriented systems are often sequential decision-making models where optimal policies are learned via reinforcement learning from a high number of user interactions [Shah et al., 2016, Dhingra et al., 2016, Liu et al., 2017, Su et al., 2017, Cuayáhuitl, 2017]. However, building such systems with limited user interactions is remarkably challenging. Thus, supervised learning approaches with modular SDS pipelines are still widely preferred when initial training data is limited, basically to bootstrap the goal-oriented conversational agents for further data collection [Sahay et al., 2019]. Statistical and neural network-based dialogue system toolkits and frameworks [Bocklisch et al., 2017, Ultes et al., 2017, Burtsev et al., 2018] are also heavily used in the academic and industrial research communities for implicit dialogue context management. A recent study named Conversation Learner [Shukla et al., 2020] describes an interactive DM via machine teaching with human-in-the-loop annotations. Although the majority of task-oriented dialogue systems require defining intents and entities, a recent work called SMCalFlow [Andreas et al., 2020] argues for a richer representation of dialogue state as a dataflow graph.

The NLU module within SDS processes user utterances as input and typically predicts the user intents and entities of interest. Long Short-Term Memory (LSTM) networks [Hochreiter and Schmidhuber, 1997] and Bidirectional LSTMs (BiLSTM) [Schuster and Paliwal, 1997] have been widely used for sequence learning tasks such as Intent Classification [Hakkani-Tur et al., 2016] and Slot Filling [Mesnil et al., 2015]. Jointly training Intent Recognition and Entity Extraction models have been explored recently [Zhang and Wang, 2016, Liu and Lane, 2016, Goo et al., 2018, Varghese et al., 2020]. Various hierarchical multi-task architectures are also proposed for these joint NLU tasks [Zhou et al., 2016, Wen et al., 2018, Okur et al., 2019, Vanzo et al., 2019], even some in multimodal context [Gu et al., 2017, Okur et al., 2020]. ?) proposed the Transformer, a novel network architecture based entirely on attention mechanisms [Bahdanau et al., 2014]. Transformer-based models usually achieve better results than RNN-based models for the NLU tasks [Sahay et al., 2019]. Bidirectional Encoder Representations from Transformers (BERT) [Devlin et al., 2018] is one of the main breakthroughs in pre-trained language representations, showing strong performance in numerous NLP tasks, including the NLU. More recently, ?) introduced the Dual Intent and Entity Transformer (DIET), a lightweight multi-task architecture for joint Intent Classification and Entity Recognition. The authors showed that DIET outperforms fine-tuning BERT for predicting intents and entities on a complex multi-domain NLU-Benchmark dataset [Liu et al., 2021] and is much faster to train.

Earlier studies have shown that data augmentation can improve the classification performance for several NLP tasks [Barzilay and McKeown, 2001, Dolan and Brockett, 2005, Lan et al., 2017, Hu et al., 2019]. Back-translation [Sennrich et al., 2016] is a popular method to construct paraphrase corpora by translating the samples to another language and then back to the original language. ?) employed the Neural Machine Translation (NMT) [Sutskever et al., 2014] approach for translating the non-English part of the parallel corpus to obtain English-to-English paraphrase pairs. Later, the authors massively scaled this approach to generate huge paraphrase corpora called ParaNMT-50M [Wieting and Gimpel, 2018]. In order to learn how to generate meaningful paraphrases, some of the previous work have utilized the autoencoders [Socher et al., 2011, Bowman et al., 2016], encoder-decoder models such as BART [Lewis et al., 2019], and NMT [Sokolov and Filimonov, 2020]. Recent studies explore data augmentation via Natural Language Generation (NLG) for few-shot intents [Xia et al., 2020] and paraphrase generation for intents and slots in task-oriented dialogue systems [Jolly et al., 2020]. Another relevant recent work [Panda et al., 2021] is an extension of a transformer-based model by ?) that works for multilingual paraphrase generation for intents and slots, even in the zero-shot settings. Several other recent works have also been exploring data augmentation with fine-tuning large language models and few-shot learning for intent classification and slot-filling tasks [Kumar et al., 2019, Kumar et al., 2020, Lee et al., 2021].

The SDS pipeline starts from recognizing user speech via Automatic Speech Recognition (ASR) module and feed the recognized text into our NLU component. We develop NLU models performing Intent Recognition and Entity Extraction to interpret user utterances. Then we pass these user intents and entities together with multimodal inputs, such as user actions and objects, into the DM component. The multimodal dialogue manager handles verbal and non-verbal communication inputs from the NLU (e.g., intents and entities) and separate external nodes processing visual and audio information (e.g., faces, poses, gestures, objects, events and actions). Note that we pass these multimodal inputs directly to the DM (bypassing the NLU) in the form of relevant multimodal intents for goal-oriented interactions. The Dialogue State Tracking (DST) model tracks what has happened (i.e., the dialogue state) within the DM. Then, the output of DST is used by the Dialogue Policy to decide which action the system should take next. Our DM models predict the appropriate agent actions and responses based on all the available contextual information (i.e., language-audio-visual inputs, game events, and dialogue history/context from previous turns). That means the DM generates sequential actions for bot utterances and non-verbal events. When the verbal response types are predicted, based on the output classes of DM, the NLG module retrieves actual bot responses, which are template-based in our use-cases. We create a variety of responses by preparing multiple templates (i.e., 3-to-6 variations) for each response type. Among these, the final response template randomly assigned at run-time. Generating grammatically correct and semantically coherent responses is challenging with such scarce datasets. Hence, this approach is more reliable than training NLG models for our application. Finally, the generated text responses are sent to the Text-to-Speech (TTS) module to output agent utterances. Non-verbal agent actions such as animations and game events are sent to the Unity application, which serves as the end User Interface (UI) displaying the agent and learning game content. Figure 2 illustrates the schematic representation of our modular SDS pipeline.

Our NLU and DM models are built on top of the Rasa open-source framework [Bocklisch et al., 2017]. The former baseline Intent Classifier in Rasa was based on supervised embeddings provided within the Rasa NLU, which is an embedding-based text classifier that embeds user utterances and intents into the same vector space. This former baseline architecture is inspired by the StarSpace work [Wu et al., 2017], where the embeddings are trained by maximizing the similarity between intents and utterances. In previous work [Sahay et al., 2019], we enriched this former baseline NLU/Intent Recognition architecture available in Rasa by incorporating additional features and adapting alternative network architectures. To be more precise, we adapted the Transformer network [Vaswani et al., 2017] and incorporated pre-trained BERT embeddings [Devlin et al., 2018] to improve the Intent Recognition performance, as shown in ?). In this current study, our new baseline NLU model is the best-performing approach from our previous work [Sahay et al., 2019], which we would call TF+BERT in our experiments.

In this work, we explore potential improvements in Intent Classification performance by adapting the recent DIET architecture [Bunk et al., 2020]. DIET is a transformer-based multi-task architecture for joint Intent Recognition and Entity Extraction. DIET can incorporate pre-trained word and sentence embeddings from language models as dense features, with the flexibility to combine these with token level one-hot encodings and multi-hot encodings of character n-grams as sparse features. Note that one can use any pre-trained embeddings as dense features in DIET, such as GloVe [Pennington et al., 2014], BERT [Devlin et al., 2018], and ConveRT [Henderson et al., 2020]. Conversational Representations from Transformers (ConveRT) is another recent and promising architecture to obtain pre-trained representations that are well-suited for Conversational AI applications, especially for the Intent Classification task. Both DIET and ConveRT are lightweight architectures with faster training capabilities than their counterparts. For all the above reasons, we adapted the DIET architecture and incorporated pre-trained ConveRT embeddings to improve our Intent Classification performance (and later explore the Entity Recognition capabilities). We would call this approach DIET+ConveRT in our experiments¹¹1Please refer to the ?) for hyperparameters, hardware specifications, and computational cost details..

Although the DM model development is beyond the scope of this work, we gradually migrated from the baseline Recurrent Embedding Domain Policy (REDP) [Vlasov et al., 2018] model, which is again inspired by the StarSpace algorithm [Wu et al., 2017] and used in our previous work [Sahay et al., 2019]. We adopted a more recent and suitable Transformer Embedding Dialogue (TED) policy [Vlasov et al., 2019] architecture, where a transformer’s self-attention mechanism operates over the sequence of dialogue turns.

Data augmentation via paraphrase generation is an effective strategy that we explored in this study to improve the Intent Classification performance, particularly when we have limited original data to train our NLU models. With that motivation, we developed a data augmentation module via training a sequence-to-sequence paraphrasing model to generate paraphrased samples from the original seed utterances to augment the NLU training data. We propose several paraphrasing-based approaches and augmentation strategies to over-sample the intent classes and investigate their effects on the NLU performance. We examine the data augmentation via paraphrasing with a few simple heuristics (i.e., paraphrasing only for the low-sample intents or minority classes or excluding the intent types with samples having shorter utterance lengths). We also investigate model-in-the-loop data augmentation techniques (i.e., augmenting only the paraphrased utterances with successful predictions and checking the confidence level thresholds using the initial NLU models trained on original samples).

For effective paraphrase generation from the seed samples, we adapted the BART sequence-to-sequence model [Lewis et al., 2019] that we fine-tuned on the back-translated English sentences from the combination of following three datasets: the Microsoft Research Paraphrase (MSRP) corpus [Dolan and Brockett, 2005], ParaNMT corpora [Wieting and Gimpel, 2018, Wieting et al., 2017], and the PAWS dataset [Zhang et al., 2019, Yang et al., 2019]. The MSRP is a corpus containing 5800 pairs of sentences extracted from web news sources, along with human annotations indicating whether each pair captures a semantic equivalence/paraphrase relationship. The ParaNMT-50M is a dataset of more than 50 million English-English sentential paraphrase pairs back-translated from the Czeng1.6 corpus [Bojar et al., 2016]. The PAWS is a corpus containing 108,463 human-labeled and well-formed paraphrase pairs. Note that we also trained paraphrasing models using GPT-2 [Radford et al., 2019] fine-tuning to augment the training set. Finally, we decided to stick with the sequence-to-sequence paraphrasing model with BART fine-tuning as it performed slightly better than GPT-2 fine-tuning version, which is expected as the BART model can be seen as a generalized BERT [Devlin et al., 2018] encoder and GPT [Radford et al., 2018] decoder.

We conduct our experiments on the KidSpace NLU datasets having utterances from multimodal math learning experiences (i.e., Planting and Watering activities) designed for 5-to-8 years-old kids [Sahay et al., 2021, Aslan et al., 2022]. These are the initial proof-of-concept (POC) datasets to bootstrap the agents to be deployed. These POC datasets are manually created based on User Experience (UX) design studies for training the SDS models and validated with UX sessions in the lab with multiple kids going through play-based learning activities. The NLU datasets have a limited number of utterances, which are manually annotated for intent types that we defined for each use-case or learning game/activity (see section 3). For the Flowerpot game, we have the Planting Flowers dataset with 1927 utterances, and for the NumberGrid activity, we have a separate Watering Flowers dataset with 2115 utterances. Some of our intents are highly generic across usages and activities (e.g., affirm, deny, next_step, out_of_scope, goodbye), whereas the rest are highly domain-dependent and task-specific (e.g., intro_meadow, answer_flowers, answer_water, ask_number, counting). Note that our dialogue system needs to process and interpret utterances received either from the kids or the adult (i.e., the Facilitator) present in the room. Therefore, almost half of our intent types are defined based on what the game flow expects from the Facilitator (e.g., to progress the games or guide the children). Table 1 shows the statistics of our NLU datasets.

To evaluate the Intent Recognition performances, the baseline NLU model that we call TF+BERT is compared with the DIET+ConveRT model that we adapted recently (see section 5.1). We conduct these evaluations on both the Planting and Watering datasets. Table 2 summarizes the Intent Classification performance results in weighted average F1-scores. We perform 10-fold cross-validation (CV) over the dataset for each run, and we report the results based on the average of 3 runs.

As we can observe from Table 2, adapting the lightweight DIET architecture [Bunk et al., 2020] with pre-trained ConveRT embeddings [Henderson et al., 2020] for our NLU models significantly improved the Intent Classification performance, which is consistent across different use-cases (i.e., Planting and Watering Flowers game datasets). With that observation, we have updated the NLU component in our multimodal SDS pipeline (see Figure 2) by replacing the TF+BERT model (i.e., previously best-performing Baseline + BERT + Transformer model in ?)) with this promising DIET+ConveRT model.

The goal here is to explore the potential benefits of paraphrasing-based data augmentation to further improve the Intent Recognition models. Our final BART-based Seq2Seq model²²2https://simpletransformers.ai/docs/seq2seq-model/ (see section 5.2) is utilized for paraphrase generation to conduct augmentation experiments. All paraphrasing experiments are conducted on the Planting Flowers dataset with a limited number of original seed utterances (i.e., less than 2K samples). We propose and investigate the following data augmentation methods with certain rule-based heuristics and model-in-the-loop (MITL) approaches:

• •

  baseline (aug3/aug5/aug10): Augment the original NLU dataset with paraphrased samples. We configured the number of samples to be generated as 3/5/10 (i.e., for each original utterance, x3/x5/x10 paraphrased samples are generated).

• •

  inc6low: Augment data only for 6 low-sample intents (i.e., having less than 50 utterances). We simply used the original plus paraphrased samples for 6 intents with fewer number of utterances (i.e., intro_meadow, help_affirm, everyone_understand, oscar_understand, ask_number, next_step), whereas we only used the original samples for rest of the 8 intents with higher number of utterances (no need for more variations for those).

• •

  exc5short: Augment data except for 5 intents with seed samples having short utterance lengths. We only used the original samples for 5 intents with short utterances (i.e., affirm, deny, answer_flowers, answer_valid, answer_invalid), whereas we used the original plus paraphrased samples for rest of the 9 intents with longer utterances (as variations help for those).

• •

  success: Augment only the paraphrased samples that are classified correctly into the same intent class as their seed samples (successful predictions). For this MITL approach, we first trained the NLU model (DIET+ConveRT) on the original/seed dataset, then classified the paraphrased samples using this initial NLU model to obtain successful predictions. We assume the paraphrased samples should belong to the same class as seed samples, and the idea is to filter out noisy synthetic samples that belong to other classes.

• •

  success_conf90: Augment only the paraphrased samples that are classified correctly into the same intent class as their seed samples (successful predictions) with a confidence score of 0.9 or higher. Another MITL approach using the same initial NLU model as in success. The confidence check ensures better noise filtering, and the threshold of 0.9 is chosen empirically after checking the confidence histograms on paraphrased samples.

• •

  all_conf90: Augment only the paraphrased samples that are classified into any intent type (regardless of their seed samples’ intent class) with a confidence score of 0.9 or higher. Another MITL approach using the same initial NLU model as in success. We removed the assumption that paraphrased samples should be of the same class as their seed samples and still augment them into the predicted class samples if confidence is high.

Figure 3 depicts an example seed utterance and its paraphrase generation outputs for x5. In all data augmentation experiments, we observed repetitions due to duplicate samples generated by the paraphraser, and we removed those duplicates from the final augmented datasets. Note that while augmenting the data, paraphrased samples are assumed to have the same intent labels as their seed samples they are generated from, except for the all_conf90 case (for which we assign the labels predicted by the initial NLU model if the confidence is 0.9 or higher).

Table 3 summarizes the Intent Recognition performances in weighted-avg F1-scores after the data augmentation with paraphrased samples. We report the NLU results on an average of 3 runs and perform a 10-fold CV over original/augmented datasets for each run. In each such fold, the 10% test partition has the original samples only, whereas the 90% training partition is augmented with the paraphrased samples. This setup ensures we evaluate the models on the same original/seed samples only, while we can expand the training data with more variations. As shown in Table 3, the baseline approach of simply augmenting the data with paraphrased samples does not help but slightly hurt the NLU performances. That is due to the possible noises in synthetic data generated via paraphraser. However, with our proposed heuristics and MITL approaches, data augmentation helps improve our NLU results. When we augment for low-sample intents only (i.e., inc6low), we start observing slight improvements and increasing the number of paraphrased samples help. We get event better jumps when we exclude short-sample intents (i.e., exc5short). Augmenting only the paraphrased samples that are classified (with high confidence) into the same label as their seed samples (i.e., success_conf90) is the best performing approach, boosting the performance by nearly 4% (compared to training on original samples). Since we properly filter out noisy synthetic data in this case, generating more paraphrases beyond x3 and x5 also helps (i.e., aug10 $>$ aug5 $>$ aug3 for success_conf90).

In addition to data augmentation, we aim to investigate the Entity Extraction and evaluate the potential improvements via entity expansion. The idea behind this is, if we can find a way to auto-extract the entities existing in the dataset, we can perform a lexical entity enrichment via ConceptNet [Liu and Singh, 2004, Speer et al., 2017] as an external Knowledge Graph (KG). Later on, we can also explore the use of lookup tables and synonyms in the dataset to create more variations via entity expansion on top of the original and paraphrased samples. With that motivation, we used a pre-trained SpaCy Entity Recognizer³³3https://spacy.io/api/entityrecognizer that performs Named Entity Recognition (NER) [McCallum and Li, 2003, Okur et al., 2016] to automatically extract the entities in our original NLU dataset (Planting Flowers). We then re-formatted the dataset with auto-tagged entities detected within utterances. We performed 3 runs of 10-fold CV on the original dataset with SpaCy NER tagged entities. We evaluated the joint Intent and Entity Recognition performances using DIET+ConveRT.

We observed that these auto-tagged entities generated via a pre-trained SpaCy NER model do not really help improve the NLU performances. We also realized that these generic named entity types are not very much relevant to our dataset. Hence, we had to define and extract more domain-specific entities, which requires a heavy task of word-level annotations. We completed these manual annotations for domain-specific entity types on the original dataset (Planting Flowers).

Table 4 summarizes our findings. Entity Recognition F1-score improved from 72.8% to 97.1% with these manual annotations, which is not surprising. Intent Classification F1-scores slightly drop when entities come into play, which aligns with the findings in ?). Although we can extract domain-specific entities pretty accurately, these token-level manual annotations are costly, even for small-size datasets. Next, we investigate auto-annotating the domain-specific entities using ConceptNet relatedness. We provide up to 6 sample values for each domain-specific entity types that we previously defined, then construct a synonym dictionary by returning the corresponding entities in the KG if the relatedness of the value is larger than an empirical threshold of 0.7. After extracting the tokens and noun chunks via SpaCy part-of-speech (POS) tagger⁴⁴4https://spacy.io/api/tagger, we automatically annotate the domain-specific entities. We do not expect this simplistic approach to work as accurately as manual annotations but want to see how much we can improve upon generic SpaCy NER tagged entities. Surprisingly, we achieved an Entity Recognition F1-score of 92.6% with the auto-annotated entities, which we believe is a very good compromise.

We believe these domain-specific entities would help us achieve lexical entity enrichment via ConceptNet as a knowledge graph on top of the original/paraphrased samples. We also believe these results are quite encouraging in our quest to make dialog systems more robust and generalizable to new intents with limited data.