Redwood: Using Collision Detection to Grow
a Large-Scale Intent Classification Dataset

As a motivating example, suppose our intent classification system has been trained on the Clinc-150 dataset Larson et al. (2019b), an intent classification dataset consisting of 150 intents.⁵⁵5In this paper, dataset names are in italics and intent names are in teletype font. Example queries are in italics and in quotes if they appear in-line. The Clinc-150 dataset includes an intent called weather, which is meant to handle weather-related queries such as “what’s the weather like today” and “tell me the weather in New York.” Suppose further that a new developer or a new team attempts to update the intent classifier with new data that contains a new intent category, such as the get_weather intent from the HWU dataset⁶⁶6Recall from Section 1 that such updates from new teams or new developers may be from routine perfective maintenance during a model’s lifetime. Liu et al. (2019). In such a scenario, there are now training data samples that overlap substantially, but that are labeled with different intents (weather vs. get_weather in this example). Thus, upon updating the model by training on HWU’s get_weather data, the predictive performance on any weather-related inference queries might be split between these two intents. This disparity can also cause unintended consequences downstream in production models, such as calls to database systems that are triggered based on the user’s intent.

Indeed, when we train a BERT classifier on the original Clinc-150 training set, the accuracy on the weather test set is 100%. When we add a HWU’s get_weather intent to Clinc-150 to create a new 151^st intent and re-train the BERT classifier, we observe an accuracy score of 60% on the weather test set. This performance drop is a symptom of having added an intent category that collides with another intent category. Such a model—which was trained on colliding intents—could cause unexpected behavior on downstream events, especially if the weather and get_weather intents trigger different business logic workflows or system responses. We note that, while in this example, the colliding weather and get_weather intent names are quite similar, other colliding pairs like Snips’ search_screening_event and MetalWOz’s movie_listings do not have lexically similar intent names, precluding straightforward string matching of intent names.

In this subsection, we formally define our collision detection problem. We first consider a scenario in which we have two intent classification datasets, $\mathcal{A}$ and $\mathcal{B}$, where $A_{i}\in\mathcal{A}$ and $B_{j}\in\mathcal{B}$ refer to specific intent categories in each. We say that intent categories $A_{i}$ and $B_{j}$ collide if there exist a sufficient number of queries in $A_{i}$ that semantically overlap with a sufficient number of queries in $B_{j}$. This semantic overlap can occur when a developer attempts to add new intent categories to a starting training dataset—when an intent classification model trained on the combined dataset $\mathcal{A}\cup\mathcal{B}$ will cause queries belonging to $A_{i}$ to be classified in $B_{j}$ (and vice versa).

As an example, suppose we have an intent classifier built from a starting dataset such as Clinc-150, which, among other things, contains a weather intent category for weather-related inquiries (cf. Section 3.1). Suppose further that we seek to grow this starting dataset by adding datapoints from a candidate dataset such as HWU (see Section 3.1, which contains a get_weather intent category). If we naïvely combine these two datasets together, a resulting intent classifier will result in some queries from the original weather category to be classified to the newly-added get_weather category because these two categories are semantically similar. Table 1 illustrates several example colliding intents and associated queries. Our approach addresses these collisions by detecting their prevalence and quantifying their impact automatically, aiding developers in improving the quality of their datasets and scope of their dialogue systems.

Because the notion of semantic overlap can differ from category to category and dataset to dataset, we observe several classes of relationships among colliding intent categories in practice. In particular, intent collisions can be simple-pairwise, transitive, or hierarchical. In the simple-pairwise case, two intents collide with each other only, and not with any other intent in either dataset. However, we also observe transitivity within intent classes. Figure 1 illustrates example utterances within intent classes a, b, and c, where all intent classes are transitively related to one another in a cycle.

Lastly, we observe non-transitive hierarchies among colliding intents. In this case, a broad intent category from one dataset can collide with two or more intent categories that do not relate to each other. Figure 2 shows a hypothetical intent class x consisting of general banking queries, including balance inquiries and transfer requests, and classes y and z consist solely of balance inquiry and transfer requests, respectively. Here, because class x is more broad than y and z, each of y and z collide with x, but y and z do not collide with each other. Our approach can help developers reveal such cases when managing datasets, and we consider these collision relationships in the creation of our Redwood dataset.

We introduce two approaches for detecting collisions: Classifier Confusion and Data Coverage.

We evaluate the effectiveness of our intent collision approaches using several indicative datasets. These datasets can be roughly grouped into three categories: (1) intent classification datasets like Clinc-150 Larson et al. (2019b), Banking-77 Casanueva et al. (2020), ACID Acharya and Fung (2020), Outlier Larson et al. (2019a), and New (this work; a corpus that was crowdsourced in a manner similar to Larson et al. (2019b) and Larson et al. (2019a)); (2) joint slot-filling and intent classification or semantic parsing datasets like ATIS Hemphill et al. (1990); Hirschman et al. (1992, 1993); Dahl et al. (1994), Snips Coucke et al. (2018), HWU Liu et al. (2019), and MTOP Li et al. (2021); and (3) turn-based dialog datasets like DSTC-8 Kim et al. (2019) and MetalWOz Lee et al. (2019). We only consider the initial queries in the turn-based DSTC-8 and MetalWOz, and discard all subsequent dialog turns.

Queries in these datasets span a wide range of topic domains, including banking and personal finance (Banking-77 and Outlier) and insurance (ACID); other datasets cover a wide array of topic domains, such as Clinc-150 and HWU, which cover smart home, automotive, travel, banking, cooking, and others. Since we are concerned with detecting colliding intents, we do not consider any slot annotations, and we use only the first turns from the multi-turn dialog datasets. In addition, we also use the Jobs640 Califf and Mooney (1997) and Talk2Car Deruyttere et al. (2019) datasets, which, although not originally designed for intent classification tasks, are categorized in a way that admit consideration as single-intent classification for our purposes. Table 2 summarizes these datasets.

Model performance on Redwood-naïve and Redwood is shown in Table 6. First, we notice that the intent classifiers perform reasonably well on the in-scope classification task, with MobileBERT classifying queries with 91% accuracy. The models also perform well on the out-of-scope task, and discriminate between in- and out-of-scope queries with AUC scores of 0.921 and 0.928 on the Clinc-150 and Vertanen (2017) out-of-scope data.

The bottom half of Table 6 presents model performance when trained and tested on Redwood-naïve. In this case, model performance is substantially worse than models trained on the carefully-crafted Redwood dataset, confirming our hypothesis from Section 3.1 that model performance suffers if trained on data with colliding intents.

We drill deeper into the impact of intent collisions on models trained on Redwood-naïve in Table 7 which charts per-intent accuracy based on the number of other intents that collide with that intent. This table groups intents based on the number of collisions, and we see that on average, intents with no collisions exhibit higher accuracy than intents with collisions. In general, colliding intents lead to degraded accuracy: intents with one or more collisions have accuracy of around 10 or more points lower than the no-collision group, with the exception of the 6-collision group. The average accuracy of the 6-collision group on Redwood-naïve is indeed surprising, and we posit that the MobileBERT model—a high-capacity transformer model—can learn the nuances of each individual intent, even if they do semantically collide.