Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropization

With a surge of interest in few-shot learning Finn et al. (2017); Vinyals et al. (2016); Snell et al. (2017), few-shot intent detection has started to receive attention. Earlier works mainly focus on model design, using capsule network Geng et al. (2019), variational autoencoder Xia et al. (a), or metric functions Yu et al. (2018); Nguyen et al. (2020). Recently, PLMs-based methods have shown promising performance in a variety of NLP tasks and become the model of choice for few-shot intent detection. Zhang et al. (c) cast few-shot intent detection into a natural language inference (NLI) problem and fine-tune PLMs on NLI datasets. Zhang et al. (b) propose to fine-tune PLMs on unlabeled utterances by contrastive learning. Zhang et al. (a) leverage a small set of public annotated intent detection benchmarks to fine-tune PLMs with standard supervised training and observe promising performance on cross-domain few-shot intent detection. Meanwhile, the study of few-shot intent detection has been extended to other settings including semi-supervised learning Dopierre et al. (b, a), generalized setting Nguyen et al. (2020), multi-label classification Hou et al. (2021), and incremental learning Xia et al. (b). In this work, we consider standard few-shot intent detection, following the setup of Zhang et al. (a) and aiming to improve supervised pre-training with isotropization.

Recent works have shown that further pre-training off-the-shelf PLMs using dialogue corpora Henderson et al. (b); Peng et al. (2020, 2021) are beneficial for task-oriented downstream tasks such as intent detection. Specifically, TOD-BERT Wu et al. (2020) conducts self-supervised learning on diverse task-oriented dialogue corpora. ConvBERT Mehri et al. (2020) is pre-trained on a $700$ million open-domain dialogue corpus. Vulić et al. (2021) propose a two-stage procedure: adaptive conversational fine-tuning followed by task-tailored conversational fine-tuning. In this work, we follow Zhang et al. (a) to further pre-train PLMs using a small amount of labeled utterances from public intent detection benchmarks.

Isotropy is a key geometric property of the semantic space of PLMs. Recent studies identify the anisotropy problem of PLMs Cai et al. (2020); Ethayarajh (2019); Mu and Viswanath (2018); Rajaee and Pilehvar (2021c), which is also known as the representation degeneration problem Gao et al. (a): word embeddings occupy a narrow cone, which suppresses the expressiveness of PLMs. To resolve the problem, various methods have been proposed, including spectrum control Wang et al. (2019), flow-based mapping Li et al. (2020), whitening transformation Su et al. (2021); Huang et al. (2021), contrastive learning Gao et al. (b), and cluster-based methods Rajaee and Pilehvar (2021a). Despite their effectiveness, these methods are designed for off-the-shelf PLMs. The interaction between isotropization and fine-tuning PLMs remains under-explored. A most recent work by Rajaee and Pilehvar shows that there might be a conflict between the two operations for the semantic textual similarity (STS) task. On the other hand, Zhou et al. (2021) propose to fine-tune PLMs with isotropic batch normalization on some supervised tasks, but it requires a large amount of training data. In this work, we study the interaction between isotropization and supervised pre-training (fine-tuning) PLMs on intent detection tasks.

Following Mu and Viswanath (2018); Biś et al. (2021), we adopt the following measurement of isotropy:

$$\begin{split}\text{I}(\mathbf{V})=\frac{\min_{\mathbf{c}~{}\in~{}C}\text{Z}(\mathbf{c},\mathbf{V})}{\max_{\mathbf{c}~{}\in~{}C}\text{Z}(\mathbf{c},\mathbf{V})},\end{split}$$

(1)

where $\mathbf{V}\in\mathbb{R}^{N\times d}$ is the matrix of stacked embeddings of $N$ utterances (note that the embeddings have zero mean), $C$ is the set of unit eigenvectors of $\mathbf{V}^{\top}\mathbf{V}$, and $\text{Z}(\mathbf{c},\mathbf{V})$ is the partition function Arora et al. (b) defined as:

$$\begin{split}\text{Z}(\mathbf{c},\mathbf{V})=\sum_{i=1}^{N}{\exp{\left({\mathbf{c}^{\top}\mathbf{v}_{i}}\right)}},\end{split}$$

(2)

where $\mathbf{v}_{i}$ is the $i_{\text{th}}$ row of $\mathbf{V}$. $\text{I}(\mathbf{V})\in\left[0,1\right]$, and 1 indicates perfect isotropy.

To observe the impact of fine-tuning on isotropy, we follow IntentBERT Zhang et al. (a) to fine-tune BERT Devlin et al. (2019) with standard supervised training on a small set of an intent detection benchmark OOS Larson et al. (2019) (details are given in Section 4.1). We then compare the isotropy of the original embedding space (BERT) and the embedding space after fine-tuning (IntentBERT) on target datasets. As shown in Table 1, after fine-tuning, the isotropy of the embedding space is notably decreased on all datasets. Hence, it can be seen that fine-tuning may render the feature space more anisotropic.

To examine the effect of isotropization on a fine-tuned model, we apply two strong isotropization techniques to IntentBERT: dropout-based contrastive learning Gao et al. (b) and whitening transformation Su et al. (2021). The former fine-tunes PLMs in a contrastive learning manner¹¹1We refer the reader to the original paper for details., while the latter transforms the semantic feature space into an isotropic space via matrix transformation. These methods have been demonstrated highly effective Gao et al. (b); Su et al. (2021) when applied to off-the-shelf PLMs, but things are different when they are applied to fine-tuned models. As shown in Fig. 2, contrastive learning improves isotropy, but it significantly lowers the performance on two benchmarks. As for whitening transformation, it has inconsistent effects on the two datasets, as shown in Fig. 3. It hurts the performance on HWU64 (Fig. 3(a)) but yields better results on BANKING77 (Fig. 3(b)), while producing nearly perfect isotropy on both. The above observations indicate that isotropization may hurt fine-tuned models, which echoes the recent finding of Rajaee and Pilehvar.

Few-shot intent detection targets to train a good intent classifier with only a few labeled data $\mathcal{D}_{\text{target}}=\{(x_{i},y_{i})\}_{N_{t}}$, where $N_{t}$ is the number of labeled samples in the target dataset, $x_{i}$ denotes the $i_{\text{th}}$ utterance, and $y_{i}$ is the label.

To tackle the problem, Zhang et al. (a) propose to learn intent detection skills (fine-tune a PLM) on a small subset of public intent detection benchmarks by supervised pre-training. Denote by $\mathcal{D}_{\text{source}}=\{(x_{i},y_{i})\}_{N_{s}}$ the source data used for pre-training, where $N_{s}$ is the number of examples. The fine-tuned PLM can be directly used on the target dataset. It has been shown that this method can work well when the label spaces of $\mathcal{D}_{\text{source}}$ and $\mathcal{D}_{\text{target}}$ are disjoint.

Specifically, the pre-training is conducted by attaching a linear layer (as the classifier) on top of the utterance representation generated by the PLM:

$$\begin{split}\text{p}(y|\mathbf{h}_{i})=\text{softmax}\left(\mathbf{W}\mathbf{h}_{i}+\mathbf{b}\right)\in\mathbb{R}^{L},\end{split}$$

(3)

where $\mathbf{h}_{i}\in\mathbb{R}^{d}$ is the representation of the $i_{\text{th}}$ utterance in $\mathcal{D}_{\text{source}}$, $\mathbf{W}\in\mathbb{R}^{L\times d}$ and $\mathbf{b}\in\mathbb{R}^{L}$ are the parameters of the linear layer, and $L$ is the number of classes. The model parameters $\theta=\left\{\phi,\mathbf{W},\mathbf{b}\right\}$, with $\phi$ being the parameters of the PLM, are trained on $\mathcal{D}_{\text{source}}$ with a cross-entropy loss:

$$\theta=\operatorname*{arg\,min}_{\theta}\mathcal{L}_{\text{ce}}\left(\mathcal{D}_{\text{source}};\theta\right).$$

(4)

After supervised pre-training, the linear layer is removed, and the PLM can be immediately used as a feature extractor for few-shot intent classification on target data. As shown in Zhang et al. (a), a parametric classifier such as logistic regression can be trained with only a few labeled samples to achieve good performance.

However, our analysis in Section 3.2 shows the limitation of supervised pre-training, which yields a anisotropic feature space.

To mitigate the anisotropy of the PLM fine-tuned by supervised pre-training, we propose a joint training objective by adding a regularization term $\mathcal{L}_{\text{reg}}$ for isotropization:

$$\mathcal{L}=\mathcal{L}_{\text{ce}}(\mathcal{D}_{\text{source}};\theta)+\lambda\mathcal{L}_{\text{reg}}(\mathcal{D}_{\text{source}};\theta),$$

(5)

where $\lambda$ is a weight parameter. The aim is to learn intent detection skills while maintaining an appropriate degree of isotropy. We devise two different regularizers introduced as follows.

Contrastive-learning-based Regularizer. Inspired by the recent success of contrastive learning in mitigating anisotropy Yan et al. (2021); Gao et al. (b), we employ the dropout-based contrastive learning loss used in Gao et al. (b) as the regularizer:

$$\mathcal{L}_{\text{reg}}=-\frac{1}{N_{b}}\sum_{i}^{N_{b}}\text{log}\frac{e^{\text{sim}(\mathbf{h}_{i},\mathbf{h}_{i}^{+})/\tau}}{\sum_{j=1}^{N_{b}}e^{\text{sim}(\mathbf{h}_{i},\mathbf{h}_{j}^{+})/\tau}}.$$

(6)

In particular, $\mathbf{h}_{i}\in\mathbb{R}^{d}$ and $\mathbf{h}_{i}^{+}\in\mathbb{R}^{d}$ are two different representations of utterance $x_{i}$ generated by the PLM with built-in standard dropout Srivastava et al. (2014), i.e., $x_{i}$ is passed to the PLM twice with different dropout masks to produce $\mathbf{h}_{i}$ and $\mathbf{h}_{i}^{+}$. $\text{sim}(\mathbf{h}_{1},\mathbf{h}_{2})$ denotes the cosine similarity between $\mathbf{h}_{1}$ and $\mathbf{h}_{2}$. $\tau$ is the temperature parameter. $N_{b}$ is the batch size. Since $\mathbf{h}_{i}$ and $\mathbf{h}_{i}^{+}$ represent the same utterance, they form a positive pair. Similarly, $\mathbf{h}_{i}$ and $\mathbf{h}_{j}^{+}$ form a negative pair, since they represent different utterances. An example is given in Fig. 4(a). By minimizing the contrastive loss, positive pairs are pulled together while negative pairs are pushed away, which in theory enforces an isotropic feature space Gao et al. (b). In Gao et al. (b), the contrastive loss is used as the single objective to fine-tune off-the-shelf PLMs in an unsupervised manner, while in this work we use it jointly with supervised pre-training to fine-tune PLMs for few-shot learning.

Correlation-matrix-based Regularizer. The above regularizer enforces isotropization implicitly. Here, we propose a new regularizer that explicitly enforces isotropization. The perfect isotropy is characterized by zero covariance and uniform variance Su et al. (2021); Zhou et al. (2021), i.e., a covariance matrix with uniform diagonal elements and zero non-diagonal elements. Isotropization can be achieved by endowing the feature space with such statistical property. However, as will be shown in Section 5, it is difficult to determine the appropriate scale of variance. Therefore, we base the regularizer on correlation matrix :

$$\mathcal{L}_{\text{reg}}=\lVert\mathbf{\Sigma}-\mathbf{I}\rVert,$$

(7)

where $\lVert\cdot\rVert$ denotes Frobenius norm, $\mathbf{I}\in\mathbb{R}^{d\times d}$ is the identity matrix, $\mathbf{\Sigma}\in\mathbb{R}^{d\times d}$ is the correlation matrix with $\mathbf{\Sigma}_{ij}$ being the Pearson correlation coefficient between the $i_{\text{th}}$ dimension and the $j_{\text{th}}$ dimension. As shown in Fig. 4(b), $\mathbf{\Sigma}$ is estimated with utterances in the current batch. By pushing the correlation matrix towards the identity matrix during training, we can learn a more isotropic feature space.

Moreover, the proposed two regularizers can be used together as follows:

$$\begin{split}\mathcal{L}=\mathcal{L}_{\text{ce}}(\mathcal{D}_{\text{source}};\theta)+\lambda_{1}\mathcal{L}_{\text{cl}}(\mathcal{D}_{\text{source}};\theta)\\ +\lambda_{2}\mathcal{L}_{\text{cor}}(\mathcal{D}_{\text{source}};\theta),\end{split}$$

(8)

where $\lambda_{1}$ and $\lambda_{2}$ are the weight parameters, and $\mathcal{L}_{\text{cl}}$ and $\mathcal{L}_{\text{cor}}$ denote CL-Reg and Cor-Reg, respectively. Our experiments show that better performance is often observed when they are used together.

Datasets. To perform supervised pre-training, we follow  Zhang et al. to use the OOS dataset Larson et al. (2019) which contains diverse semantics of $10$ domains. Also following  Zhang et al., we exclude the domains “Banking” and “Credit Cards” since they are similar in semantics to one of the test dataset BANKING77. We then use $6$ domains for training and $2$ for validation, as shown in Table 2. For evaluation, we employ three datasets: BANKING77 Casanueva et al. (2020) is an intent detection dataset for banking service. HINT3 Arora et al. (a) covers $3$ domains, “Mattress Products Retail”, “Fitness Supplements Retail”, and “Online Gaming”. HWU64 Liu et al. (2019a) is a large-scale dataset containing $21$ domains. Dataset statistics are summarized in Table 3.

Our Method. Our method can be applied to fine-tune any PLM. We conduct experiments on two popular PLMs, BERT Devlin et al. (2019) and RoBERTa Liu et al. (2019b). For both of them, the embedding of $[CLS]$ is used as the utterance representation in Eq. 3. We employ logistic regression as the classifier. We select the hyperparameters $\lambda,\lambda_{1},\lambda_{2}$, and $\tau$ by validation. The best hyperparameters are provided in Table 4.

Baselines. We compare our method to the following baselines. First, for BERT-based methods, BERT-Freeze freezes BERT; CONVBERT Mehri et al. (2020), TOD-BERT Wu et al. (2020), and DNNC-BERT Zhang et al. (c) further pre-train BERT on conversational corpus or natural language inference tasks. USE-ConveRT Henderson et al. (a); Casanueva et al. (2020) is a transformer-based dual-encoder pre-trained on conversational corpus. CPFT-BERT is the re-implemented version of CPFT Zhang et al. (b), by further pre-training BERT in an unsupervised manner with mask-based contrastive learning and masked language modeling on the same training data as ours. IntentBERT Zhang et al. (a) further pre-trains BERT via supervised pre-training described in Section 4.1. To guarantee a fair comparison, we provide IntentBERT-ReImp, the re-implemented version of IntentBERT, which uses the same random seed, training data, and validation data as our methods. Second, for RoBERTa-based baselines, RoBERTa-Freeze freezes the model. WikiHowRoBERTa Zhang et al. (d) further pre-trains RoBERTa on synthesized intent detection data. DNNC-RoBERTa and CPFT-RoBERTa are similar to DNNC-BERT and CPFT-BERT except the PLM. IntentRoBERTa is the re-implemented version of IntentBERT based on RoBERTa, with uses the same random seed, training data, and validation data as our method. Finally, to show the superiority of the joint fine-tuning and isotropization, we compare our method against whitening transformation Su et al. (2021). BERT-White and RoBERTa-White apply the transformation to BERT and RoBERTa, respectively. IntentBERT-White and IntentRoBERTa-White apply the transformation to IntentBERT-ReImp and IntentRoBERTa, respectively.

All baselines use logistic regression as classifier except DNNC-BERT and DNNC-RoBERTa, wherein we follow the original work²²2https://github.com/salesforce/DNNC-few-shot-intent to train a pairwise encoder for nearest neighbor classification.

Training Details. We use PyTorch library and Python to build our model. We employ Hugging Face implementation³³3https://github.com/huggingface/transformers of bert-base-uncased and roberta-base. We use Adam Kingma and Ba (2015) as the optimizer with learning rate of $2e-05$ and weight decay of $1e-03$. The model is trained with Nvidia RTX 3090 GPUs. The training is early stopped if no improvement in validation accuracy is observed for $100$ steps. The same set of random seeds, $\left\{1,2,3,4,5\right\}$, is used for IntentBERT-ReImp, IntentRoBERTa, and our method.

Evaluation. The baselines and our method are evaluated on $C$-way $K$-shot tasks. For each task, we randomly sample $C$ classes and $K$ examples per class. The $C\times K$ labeled examples are used to train the logistic regression classifier. Note that we do not further fine-tune the PLM using the labeled data of the task. We then sample another $5$ examples per class as queries. Fig. 1 gives an example with $C=2$ and $K=1$. We report the averaged accuracy of $500$ tasks randomly sampled from $\mathcal{D}_{\text{target}}$.

The main results are provided in Table 5 (BERT-based) and Table 6 (RoBERTa-based). The following observations can be made. First, our proposed regularized supervised pre-training, with either CL-Reg or Cor-Reg, consistently outperforms all the baselines by a notable margin in most cases, indicating the effectiveness of our method. Our method also outperforms whitening transformation, demonstrating the superiority of the proposed joint fine-tuning and isotropization framework. Second, Cor-Reg slightly outperforms CL-Reg in most cases, showing the advantage of enforcing isotropy explicitly with the correlation matrix. Finally, CL-Reg and Cor-Reg show a complementary effect in many cases, especially on BANKING77. The above observations are consistent for both BERT and RoBERTa. It can be also seen that higher performance is often attained with RoBERTa.

The observed improvement in performance comes with an improvement in isotropy. We report the change in isotropy by the proposed regularizers in Table 7. It can be seen that both regularizers and their combination make the feature space more isotropic compared to IntentBERT-ReImp that only uses supervised pre-training. In addition, in general, Cor-Reg can achieve better isotropy than CL-Reg.

Moderate isotropy is helpful. To investigate the relation between the isotropy of the feature space and the performance of few-shot intent detection, we tune the weight parameter $\lambda$ of Cor-Reg to increase the isotropy and examine the performance. As shown in Fig. 5, a common pattern is observed: the best performance is achieved when the isotropy is moderate. This observation indicates that it is important to find an appropriate trade-off between learning intent detection skills and learning an insotropic feature space. In our method, we select the appropriate $\lambda$ by validation.

Correlation matrix is better than covariance matrix as regularizer. In the design of Cor-Reg (Section 4.2), we use the correlation matrix, rather than the covariance matrix, to characterize isotropy, although the latter contains more information – variance. The reason is that it is difficult to determine the proper scale of the variances. Here, we conduct experiments using the covariance matrix, by pushing the non-diagonal elements (covariances) towards $0$ and the diagonal elements (variances) towards $1$, $0.5$, or the mean value, which are denoted by Cov-Reg-1, Cov-Reg-0.5, and Cov-Reg-mean respectively in Table 8. It can be seen that all the variants perform worse than Cor-Reg.

Our method is complementary with batch normalization. Batch normalization Ioffe and Szegedy (2015) can potentially mitigate the anisotropy problem via normalizing each dimension with unit variance. We find that combining our method with batch normalization yields better performance, as shown in Table 9.

The performance gain is not from the reduction in model variance. Regularization techniques such as L1 regularization Tibshirani (1996) and L2 regularization Hoerl and Kennard (1970) are often used to improve model performance by reducing model variance. Here, we show that the performance gain of our method is ascribed to the improved isotropy (Table 7) rather than the reduction in model variance. To this end, we compare our method against L2 regularization with a wide range of weights, and it is observed that reducing model variance cannot achieve comparable performance to our method, as shown in Fig. 6.

The computational overhead is small. To analyze the computational overheads incurred by CL-Reg and Cor-Reg, we decompose the duration of one epoch of our method using the two regularizers jointly. As shown in Fig. 7, the overheads of CL-Reg and Cor-Reg are small, only taking up a small portion of the time.