Low-Resource Task-Oriented Semantic Parsing
via Intrinsic Modeling

Task-oriented conversational assistants face an increasing demand to support a wide range of domains (e.g., reminders, messaging, weather) as a result of their emerging popularity Chen et al. (2020); Ghoshal et al. (2020). For practitioners, enabling these capabilities first requires training semantic parsers which map utterances to frames executable by assistants Gupta et al. (2018); Einolghozati et al. (2018); Pasupat et al. (2019); Aghajanyan et al. (2020); Li et al. (2020); Chen et al. (2020); Ghoshal et al. (2020). However, current methodology typically requires crowdsourcing thousands of samples for each domain, which can be both time-consuming and cost-ineffective at scale Wang et al. (2015); Jia and Liang (2016); Herzig and Berant (2019); Chen et al. (2020). One step towards reducing these data requirements is improving the sample efficiency of current, de facto copy-generate parsers. However, even when leveraging pre-trained transformers, these models are often ill-equipped to handle low-resource settings, as they fundamentally lack inductive bias and cross-domain reusability.

In this work, we explore a task-oriented semantic parsing model which leverages the intrinsic properties of an ontology to improve generalization. To illustrate, consider the ontology label SL:TIME_ZONE : a slot representing the time zone in a user’s query. Copy-generate models typically treat this label as a discrete entity, relying on parallel data to extrinsically learn its semantics. In contrast, our model exploits what we intrinsically know about this label, such as its categorical type (e.g., “slot”) and language-based span (e.g., “time zone”). Guided by this principle, we extract the properties of each label in a domain’s ontology, building a component with these properties and inserting each component into an inventory. By processing this domain-specific inventory through strong language models, we effectively synthesize an inductive bias useful in low-resource settings.

Concretely, we build our model on top of seq2seq, pre-trained transformer, namely BART Lewis et al. (2020), and fine-tune it to map utterances to frames, as depicted in Figure 1. Our model operates in two stages: (1) the encoder inputs a domain-specific utterance and inventory and (2) the decoder outputs a frame composed of utterance and ontology tokens. Here, instead of performing vocabulary augmentation, the standard way of “generating” ontology tokens in copy-generate parsers, we treat ontology tokens as pointers to inventory components. This is particularly useful in low-resource settings; our model is encouraged to represent ontology tokens as a union of its intrinsic properties, and as a result, does not require many labeled examples to achieve strong performance.

We view sample efficiency as one of the advantages of our approach. As such, we develop a comprehensive low-resource benchmark derived from TOPv2 Chen et al. (2020), a task-oriented semantic parsing dataset spanning 8 domains. Using a leave-one-out setup on each domain, models are fine-tuned on a high-resource, source dataset (other domains;  100K+ samples), then fine-tuned and evaluated on a low-resource, target dataset (this domain; 10-250 samples). We also randomly sample target subsets to make the transfer task more difficult. In aggregate, our benchmark provides 32 experiments, each varying in domain and number of target samples.

Both coarse-grained and fine-grained experiments show our approach outperforms baselines by a wide margin. Overall, when averaging across all domains, our inventory model outperforms a copy-generate model by +15 EM absolute (44% relative) in the most challenging setting, where only 10 target samples are provided. Notably, our base inventory model (139M parameters) also outperforms a large copy-generate model (406M parameters) in most settings, suggesting usability in resource-constrained environments. We also show systematic improvements on a per-domain basis, even for challenging domains with high compositionality and ontology size, such as reminders and navigation. Finally, through error analysis, we show our model’s predicted frames are largely precise and linguistically consistent; even when inaccurate, our frames not require substantial modifications to achieve gold quality.

Task-oriented semantic parsers typically cast parsing as transduction, utilizing seq2seq transformers to map utterances to frames comprised of intents and slots Aghajanyan et al. (2020); Li et al. (2020); Chen et al. (2020); Ghoshal et al. (2020). Because frames are a composition of utterance and ontology tokens, these models are often equipped with copy-generate mechanisms; at each timestep, the decoder either copies from the utterance or generates from the ontology See et al. (2017); Aghajanyan et al. (2020); Chen et al. (2020); Ghoshal et al. (2020). These parsers are empirically effective in high-resource settings, achieving state-of-the-art performance on numerous benchmarks Aghajanyan et al. (2020), but typically lack inductive bias in low-resource settings.

To illustrate, consider a hypothetical domain adaptation scenario where a copy-generate parser adapts to the weather domain. In standard methodology, a practitioner augments the decoder’s vocabulary with weather ontology labels, then fine-tunes the parser on weather samples. This subsequently trains the copy-generate mechanism to generate these labels as deemed appropriate. But, the efficacy of this process scales with the amount of training data as these ontology labels (though, more specifically, their embeddings) iteratively derive extrinsic meaning. Put another way, before training, there exists no correspondence between an ontology label (e.g., SL:LOCATION) and an utterance span (e.g., “Menlo Park”). Such an alignment is only established once the model has seen enough parallel data where the two co-occur.

In contrast, we focus on reducing data requirements by exploiting the intrinsic properties of ontology labels. These labels typically have several core elements, such as their label types and spans. For example, by teasing apart SL:LOCATION, we can see it is composed of the type “slot” and span “location”. These properties, when pieced together and encoded by strong language models, provide an accurate representation of what the ontology label is. While these properties are learned empirically with sufficient training data, as we see with the copy-generate parser, our goal is to train high-quality parsers with as little data as possible by explicitly supplying this information. Therefore, a central question of our work is whether we can build a parser which leverages the intrinsic nature of an ontology space while retaining the flexibility of seq2seq modeling; we detail our approach in the next section.

Illustrated in Figure 1, we develop a seq2seq parser which uses inventories—that is, tables enumerating the intrinsic properties of ontology labels—to map utterances to frames. Inventories are domain-specific, and each component carries the intrinsic properties of a single label in the domain’s ontology. On the source-side, a pre-trained encoder consumes both an utterance and inventory (corresponding to the utterance’s domain). Then, on the target-side, a pre-trained decoder mimics a copy-generate mechanism by either selecting from the utterance or ontology. Instead of selecting ontology labels from an augmented vocabulary, as in copy-generate methods, our decoder naturally references these labels in the source-side inventory through self-attention. The sequence ultimately decoded during generation represents the frame.

As alluded to earlier, we focus on two intrinsic properties of ontology labels: types and spans. The type is particularly useful for enforcing syntactic structure; for example, the rule “slots cannot be nested in other slots” Gupta et al. (2018) is challenging to meet unless a model can delineate between intents and slots. Furthermore, the span is effectively a natural language description, which provides a general overview of what the label aligns to. Though inventory components can incorporate other intrinsic properties, types and spans do not require manual curation and can be automatically sourced from existing annotations.

Despite the reformulation of the semantic parsing task with inventories, our approach inherits the flexibility and simplicity of copy-generate models Aghajanyan et al. (2020); Li et al. (2020); Chen et al. (2020); Ghoshal et al. (2020); we also treat parsing as transduction, leverage pre-trained modules, and fine-tune with log loss. However, a key difference is our parser is entirely text-to-text and does not require extra parameters, which we show promotes reusability in low-resource settings. In the following sub-sections, we elaborate on our approach in more detail and comment on several design decisions.

In this section, we describe our low-resource benchmark used to assess the sample efficiency of our model. The benchmark is derived from TOPv2 Chen et al. (2020), a task-oriented semantic parsing dataset covering 8 domains: alarm, event, messaging, music, navigation, reminder, timer, and weather. TOPv2 samples have a combination of both linear and nested frames, uniquely reflecting the data distribution our parsers are likely to encounter in practice.

We seek to answer three questions in our experiments: (1) How sample efficient is our model when benchmarked on TOPv2-DA? (2) Does our model perform well on average or does it selectively work on particular domains? (3) How do the intrinsic components of an inventory component (e.g., types and spans) contribute to performance?

Thus far, we have demonstrated the efficacy of inventory parsers, but we have not yet conducted a thorough investigation of their errors. Though models may not achieve perfect EM in low-resource settings, they should ideally fail gracefully, making mistakes which roughly align with intuition. In this section, we assess this by combing through our model’s cross-domain errors. Using Inventory${}_{\textrm{LARGE}}$ models fine-tuned in each domain’s 1 SPIS setting, we first manually inspect 100 randomly sampled errors to build an understanding of the error distribution. Then, for each domain, we select one representative error, and present the predicted and gold frame in Table 7.

In most cases, the edit distance between the predicted and gold frames is quite low, indicating the frames our models produce are fundamentally good and do not require substantial modification. We do not see evidence of erratic behavior caused by autoregressive modeling, such as syntactically invalid frames or extraneous subword tokens in the output sequence. Instead, most errors are relatively benign; we can potentially resolve them with rule-based transformations or data augmentation, though these are outside the scope of our work. Below, we comment on specific observations:

Prior work improving the generalization of task-oriented semantic parsers can be categorized into two groups: (1) contextual model architectures and (2) fine-tuning and optimization. We compare and contrast our work along these two axes below.

In this work, we present a seq2seq-based, task-oriented semantic parser based on inventories, tabular structures which capture the intrinsic properties of an ontology space, such as label types (e.g., “slot”) and spans (e.g., “time zone”). Our approach is both simple and flexible: we leverage out-of-the-box, pre-trained transformers with no modification to the underlying architecture. We chiefly perform evaluations on TOPv2-DA, a benchmark consisting of 32 low-resource experiments across 8 domains. Experiments show our inventory parser outperforms classical copy-generate parsers by a wide margin and ablations illustrate the importance of types and spans. Finally, we conclude with an error analysis, providing insight on the types of errors practitioners can expect when using our model in low-resource settings.