Low-Resource Compositional Semantic Parsing with Concept Pretraining

Voice assistants such as Alexa, Siri, and Google Assistant often rely on semantic parsing to understand requests made by their users. The underlying semantic parsing model converts natural language user utterances into logical forms consisting of actions requested by the user (play music, check weather), called intents, and relevant entities in the request (which song? which location?), called slots. The model is built to process requests in a fixed set of domains, such as music, weather, shopping, and so on. With voice assistants increasingly pervading more aspects of daily life, systems need to be continuously updated to comprehend new intents and slots across an ever-growing number of domains.

Current semantic parsing models are trained on large amounts of annotated data from a predetermined set of domains. Extending these models to learn new intents or slots typically involves collecting and annotating large amounts of new data. This process is expensive and time-consuming. To combat this problem, researchers have proposed semantic parsing models that can be efficiently trained with fewer examples (few-shot) from new domains (Shrivastava et al., 2021; Mansimov and Zhang, 2021; Ghoshal et al., 2020; Shin et al., 2021; Desai et al., 2021; Rongali et al., 2022; Shrivastava et al., 2022). While these methods facilitate few-shot learning, they have limitations. Some of them rely on hand-crafted knowledge such as intermediate grammars or logical form templates (Shrivastava et al., 2022; Shin et al., 2021; Rongali et al., 2022). Others rely on very large pretrained language models, such as GPT-3, to perform in-context learning by appending test examples with instructional prompts (Shin et al., 2021).

In this work, we explore few-shot domain adaptation for semantic parsing without any additional hand-crafted knowledge apart from the intent and slot tag names, and with much smaller architectures that can perform efficient inference in practical production environments. We also explore zero-shot domain adaptation, when we have no annotated training data from a new domain.

To that end, we propose Concept-Seq2Seq, a novel architecture based on a state-of-the-art semantic parsing model, Seq2Seq-Ptr (Rongali et al., 2020), which uses seq2seq models and a pointer generator network to decode the target semantic parse. We augment this model with a concept encoder that encodes intents and slots from the schema and uses those encodings to conditionally decode the semantic parse. Figure 1 shows the architecture of our proposed model. We train this model on annotated data from the given domains. During inference, we simply encode all intents and slots from the schema, including new, unseen ones, into the learned concept space, and decode the target parse. This model has the same time complexity as the original Seq2Seq-Ptr model but comes with the added benefit of now being able to effectively parse utterances from unseen domains without any additional effort.

There have been a few zero-shot semantic parsing approaches proposed in the past but they either covered only simple slot-filling style utterances (Bapna et al., 2017; Lee and Jha, 2019) or compositional utterances that also came with carefully crafted intermediate representations and context-free grammars Herzig and Berant (2018); Wu et al. (2021). Our model is capable of performing zero-shot domain adaptation for compositional semantic parsing, producing meaning representations with nested intents and slots, but also doesn’t require any grammars, whose construction effort often exceeds the effort required to annotate a few examples.

In few-shot scenarios, we fine-tune our zero-shot model checkpoints further on the small number of available examples. Due to the presence of the concept encoder in our architecture, we expect to receive better knowledge-transfer advantages by encoding intent and slot tags from new domains as opposed to initializing them as new tags. To further improve performance, we propose a novel decoder-focused pretraining scheme for Concept-Seq2Seq using an entity-centric processed version of Wikidata (Vrandečić and Krötzsch, 2014) called WikiWiki (Li et al., 2022), to help it better encode unseen concepts and parse effectively.

We report the first zero-shot performance numbers for semantic parsing on the compositional TOPv2 dataset Chen et al. (2020) and show that Concept-Seq2Seq achieves commendable zero-shot performance on the flat-entity SNIPS dataset Coucke et al. (2018). We also evaluate in few-shot settings and show that we match or outperform previous state-of-the-art models while still being production-viable.

In summary, our contributions are as follows.

• •

  We propose Concept-Seq2Seq, a bi-tower architecture with a seq2seq model and a concept encoder, that can perform few-shot and zero-shot domain adaptation for compositional semantic parsing without additional handcrafted knowledge.

• •

  We propose a novel decoder-focused pretraining scheme for Concept-Seq2Seq using Wikidata that helps it better encode unseen concepts and parse effectively.

• •

  We report few-shot and zero-shot semantic parsing results on the TOPv2 and SNIPS datasets and show that our model outperforms or matches previously proposed approaches on a variety of few-shot settings.

In this section, we describe our proposed model, Concept-Seq2Seq, for low resource (few-shot and zero-shot) domain adaptation for semantic parsing. It is based on the Seq2Seq-Ptr model from Rongali et al. (2020), consisting of a sequence-to-sequence encoder-decoder component, augmented with a pointer generator network to constrain the target decoding vocabulary. Since our task at hand is to perform potential zero-shot semantic parsing with just descriptive metadata about the new domain, we modify the architecture of Seq2Seq-Ptr to incorporate information about new intents and slots from new domains by adding a concept encoder. Section 2.2 describes this architecture in detail. To help our model learn to parse utterances from unseen domains better, we also propose a novel pretraining scheme to incorporate general concept parsing knowledge into it. Section 2.3 describes this concept pretraining scheme. Finally, we describe Concept-Seq2Seq model specifics for few-shot and zero-shot settings in Section 2.4. Before we get to these sections, we first describe the source and target sequence formulation for the semantic parsing task below.

We evaluate Concept-Seq2Seq on few-shot and zero-shot domain adaptation using two popular English task-oriented semantic parsing datasets - TOPv2 (Chen et al., 2020) and SNIPS (Coucke et al., 2018). Both datasets have utterances grouped into multiple domains; TOPv2 has eight domains and SNIPS has seven intents from seven different areas, which we consider domains. TOPv2 is a large dataset consisting of 10k-20k training and 3k-7k test examples per domain. It is also comprised of compositional examples with nested intents and slots. We exclude the unsupported utterances from the training and test sets in TOPv2 for the zero-shot experiments (we use the full sets in few-shot). Unsupported utterances consist of utterances that belong to a domain but are not supported, which is impossible to learn in zero-shot. SNIPS is a smaller and simpler dataset with flat, disjoint slots. It has 2k training and 100 test examples per domain.

For zero-shot, we use a leave-one-out approach where given $n$ domains, we train models on annotated data from $n-1$ of them and evaluate on utterances from the left-out domain. For few-shot settings, where the model has access to a few annotated examples from the left-out domain, we further fine-tune using 1, 5, and 25 samples-per-intent/slot (SPIs), which we randomly sample from the training data of the left-out domain. We fine-tune Concept-Seq2Seq on three different randomly sampled training sets per domain per SPI setting and report the average performance score of the three runs.

We use a transformer encoder, initialized from a roberta-base checkpoint, for Concept-Seq2Seq. The decoder is a transformer decoder initialized from scratch and it contains 6 layers, 8 heads, and a hidden state size of 768. The concept encoder is also a transformer encoder and it is initialized from a bert-base-uncased checkpoint. We choose a BERT-based model here since it is pretrained to compute a vector for the whole sequence using the CLS token, which is what we need for encoding a concept consisting of a multi-word description. We also choose all base-size components to keep the overall model size small and expect the relative improvements shown by our model to generalize.

We train our zero-shot models using sequence cross entropy loss. We use the Adam optimizer with learning rate $2e^{-5}$ and $\epsilon=1e^{-8}$, warm-up proportion $0.1$, weight decay $0.01$, and batch-size 128. The number of epochs is set to 100 and we evaluate after every epoch and early stop with a patience of 5. For the Wikiwiki pretraining step, we use the same hyper-parameters but stop the model training after 2 epochs on the entire Wikiwiki dataset. We did not perform explicit hyper-parameter tuning.

For the few-shot experiments, we take the zero-shot model trained by excluding the given few-shot domain and fine-tune it on small set of annotated examples for 1000 epochs, evaluating after every 25 epochs. All other parameters are set to the same values as in the initial zero-shot training. We use a multiplier of 0.1 for the augmented loss from the random known-domain data batch. To speed up evaluation during training, we use teacher-forced sequence accuracy as our validation metric, which doesn’t require us to perform any beam search. During inference, we use beam search decoding with a beam size of 4.

We report exact match (EM) accuracy for the few-shot experiments, meaning the entire predicted parse has to match exactly with the gold parse. For zero-shot, we report both EM and F1 score since the task is more difficult and the performance is generally lower. At these lower numbers, F1 score, which awards partial credit to correctly tagged spans provides a better picture for improvements than EM accuracy, which requires the entire predicted parse to be correct for credit. For comparison wherever applicable, we use prior state-of-the-art models as baselines. In addition, we also report the performance of a vanilla Seq2Seq-Ptr model without concept pretraining.

In this section, we report and discuss the performance of Concept-Seq2Seq on zero-shot and few-shot domain adaptation for semantic parsing. We first briefly describe our findings in zero-shot setting and then describe findings in a variety of few-shot scenarios.

Zero-shot domain adaptation for task-oriented semantic parsing has been previously explored for simple flat queries with single intents and disjoint, non-overlapping slots. Bapna et al. (2017) and Lee and Jha (2019) encode the lexical tag features and create a token-tagging schema to create the final semantic parses. Yu et al. (2021) solve the task using a retrieve-and-fill mechanism. Our baseline model for simple queries is based on these approaches.

For complex utterances with nested structures, zero-shot semantic parsing has been explored using intermediate, concept-agnostic logical forms (Herzig and Berant, 2018; Dong and Lapata, 2018; Reddy et al., 2017) or natural language canonical forms (Wu et al., 2021). These approaches apply to semantic parsing datasets which have context free grammars and specified rules, such as database or knowledge graph queries. The effort to craft these grammars for task-oriented semantic parsing in a voice assistant setting could quite possibly be greater than annotating utterances.

A more relevant class of approaches for this work are ones that solve task-oriented semantic parsing for complex utterances in a few-shot setting using lexical tag features. Shrivastava et al. (2021) and Mansimov and Zhang (2021) modify the seq2seq architecture from Rongali et al. (2020) to perform non-autoregressive style decoding and show that their models perform better in a few-shot setting. Ghoshal et al. (2020) use adaptive label smoothing, a model-agnostic technique. Shin et al. (2021) proposed a prompting-style approach where custom instructional prompts filled with handful of annotated examples and an unsolved utterance are fed as input to GPT-3 to directly produce a semantic parse. Their approach is extremely slow and cannot be easily adapted into a zero-shot framework. Shrivastava et al. (2022) explore a retrieve-and-fill style approach where they retrieve the best scenario, an intermediate logical form consisting of the semantic frame and abstracted out tags, from a scenario bank of all supported semantic parses. Their approach is contingent on the availability of this scenario bank which could possibly entail more effort than annotating utterances. Mueller et al. (2022) and Desai et al. (2021) use lexical features from intent and slot names to create an inventory and use it as input to train semantic parsers for new domains. Mueller et al. (2022) also pretrain their model to improve generalizability but only evaluate it on an intent classification task. Desai et al. (2021) evaluate their model for full sequences and our model is similar to theirs. However, we use our inventory to create custom decoder embeddings in a seq2seq model, which removes any input size issues that their model will encounter with large inventories. We also pretrain our model with Wikidata and evaluate it in a completely zero-shot setting, in addition to few-shot. Zhao et al. (2022) is another recent question-answering-based approach that uses lexical features from the intent and slot tags by using them as context and posing questions but it has a similar input size issue with large inventories.

We propose a model called Concept-Seq2Seq to perform low-resource domain adaptation for compositional semantic parsing. Our model is built on the Seq2Seq-Ptr framework and is augmented with a concept encoder to transfer knowledge and encode unseen intents and slots from new domains through their text definitions. We also propose a novel concept pretraining scheme to incorporate general concept knowledge into our model using an entity-centric Wikipedia dataset called Wikiwiki.

We evaluate our model in zero-shot and multiple few-shot settings on Facebook TOPv2 and SNIPS datasets. We show that our model is capable of performing zero-shot domain adaptation on some domains of the TOPv2 dataset and beats a strong slot-filling baseline on the SNIPS dataset. In few-shot, over multiple dataset sizes of 1, 5, and 25 SPIs, we show that our model outperforms many strong prior models on TOPv2. Using the SNIPS dataset, we also demonstrate how our model catches up to a fully-trained semantic parsing model using just 25 SPIs on most domains. Our model is capable of low-resource domain adaptation in both compositional and flat parsing settings, without additional hand-crafted information, and with the inference behavior of a vanilla Seq2Seq-Ptr model.

This work was supported in part by Amazon Alexa AI, in part by the Chan Zuckerberg Initiative, in part by IBM Research AI through the AI Horizons Network, and in part by the National Science Foundation (NSF) grants IIS-1763618 and IIS-1955567. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.

Our models were trained on GPUs that had at least 20GB on-board memory since in addition to the traditional encoder and decoder components in a seq2seq model, we also train a concept encoder, which is around the same size as the encoder. This eliminates the use of popular GPUs such as 1080-ti and 2080-ti, unless parameter freezing or other tricks are employed during training. During inference however, once the concepts are trained, we can simply encode the target tags and this reduces to the size and performance of a traditional Seq2Seq-Ptr model.

We also report all results on models (ours and baselines) with base-size components such as roberta-base. We do this since these models are more likely to be used in production than the much bigger large-size models. Results and comparison with large-sized models is missing from this work (we expect the trends shown to generalize) and we leave this to future work.

Finally, to simulate our low-resource experiments, we randomly sample a few examples from the existing training datasets. While this is useful for experimentation, it doesn’t truly mimic a real low-resource workflow where these few examples could be carefully crafted by developers to ensure better semantic coverage in terms of the language of the utterances. This work doesn’t include any analysis on the influence of the content of the few selected examples; it just focuses on their number.