Description-Enhanced Label Embedding Contrastive Learning for Text Classification

With the full usage of large corpus and multi-layer transformers, PLMs (e.g., BERT [21]) have accomplished much progress in many NLP tasks. Thus, we take pre-trained BERT as an example to describe how to generate sentence semantic representations for the input. Moreover, inspired by ELMo [51], we also use the weighted sum of all the hidden states of words from different transformer layers as the final contextual representations of input words in sentences.

Specifically, we first split input text into BPE tokens [52] and add special token “[CLS]” at the beginning and the end of word sequence. “[SEP]” token is also considered when there is more than one sentence. Then, BERT is utilized to generate word representation $\bm{H}$ and sentence representations $\bm{v}_{g}$. As illustrated in Fig. 2(B), suppose there are $L$ layers in BERT. Contextual word representations for input text sequence $\bm{X}$ is then a pre-layer weighted sum of transformer block output, with weights $\alpha_{1},\alpha_{2},...,\alpha_{L}$.

where $\bm{h}_{0}^{l}$ denotes the representation of first token “[CLS]” at the $l^{th}$ layer, and $\bm{v}_{g}$ denotes the global semantic representation of the input. $\bm{H}$ represents the sequence features of the whole input. $\alpha_{l}$ is the weight of the $l^{th}$ layer in BERT and will be learned during model training.

The semantic relation within the text sequence is not only connected with the important words, but also affected by the local information (e.g., phrase and local structure). Though BERT leverages multi-layer transformers to perceive important words in the sentence pair, it still has some weaknesses in modelling local information. To alleviate these shortcomings, we develop a CNN-based local encoder to extract local information from the input.

Fig. 2(C) reports the local encoding structure. The input of this module is $\bm{H}$ from global encoding. We use convolution operations with different composite kernels (e.g., bi-gram and tri-gram) to process these features. Each operation with different kernels can model local patterns with different sizes. Next, we leverage average pooling and max pooling to enhance these local features and concatenate them before sending them to a non-linear transformation. Suppose we have $K$ different kernel sizes. This process can be formulated as follows:

where $CNN_{k}$ denotes the convolution operation with the $k^{th}$ kernel. $[\cdot;\cdot]$ is the concatenation operation. $\bm{v}_{l}$ represents local semantic representation of the input. $\{\bm{W}\in\mathcal{R}^{d_{p}*(2K*d_{p})},\bm{b}\in\mathcal{R}^{d_{p}}\}$ are trainable parameters. $d_{p}$ is output size of pre-trained model. $ReLu(\cdot)$ is the activation function.

After getting the global representation $\bm{v}_{g}$ and local representation $\bm{v}_{l}$, we investigate different fusion methods to integrate them, including simple concatenation, weighted concatenation, as well as weighted sum. Finally, we obtain that simple concatenation is flexible and can achieve comparable performance without extra training parameters. Thus, we employ concatenation $\bm{v}=[\bm{v}_{g};\bm{v}_{l}]$ as the final semantic representation of input text sequence.

This component is adopted to predict the label of input text, which is an essential text classification part. To be specific, the input of this component is semantic representation $\bm{v}$. We leverage a two-layer MLP to make the final classification, which can be formulated as follows:

Inspired by SSL methods in PLMs (e.g., MLM and NSP in BERT), we intend R²-Net to make full use of label information in a similar way. Therefore, we develop a novel self-supervised R² classification task. Instead of just identifying the most suitable label, we plan to obtain more information about the input text sequence by analyzing the pairwise relation between semantic representations (i.e., $\bm{v}_{1}$ for $\bm{X}_{1}$, and $\bm{v}_{2}$ for $\bm{X}_{2}$). Since a learnable nonlinear transformation between representations and loss substantially improves the model performance [44], we first transfer $\bm{v}_{1}$ and $\bm{v}_{2}$ with a nonlinear transformation. Then, we leverage heuristic matching [53, 54] to model the similarity and difference between $\bm{v}_{1}$ and $\bm{v}_{2}$. Next, we send the matching result $\bm{u}$ to an MLP with one hidden layer for final classification. This process is formulated as follows:

where concatenation can retain all the information [54]. The element-wise product is a certain measure of “similarity” between two sentences [55]. Their differences can capture the degree of distributional inclusion in each dimension [56]. $\hat{y}\in\{1,0\}$ indicates whether two input sequences have the same relation.

Apart from leveraging R² task to learn inter-class relation information, we also intend to learn intra-class connections from triplet-based relation. Thus, we introduce triplet loss [57] into R²-Net. As a fundamental similarity function, triplet loss is widely applied in information retrieval [7], and is able to pull together input sequences with the same label and push apart these with different labels. The inputs of this component are three semantic representations: $\bm{v}_{a}$ for anchor text $\bm{X}_{a}$, $\bm{v}_{p}$ for positive text $\bm{X}_{p}$, $\bm{v}_{n}$ for negative text $\bm{X}_{n}$. We first transform them into a common space with a full connection layer [44]. Then, we calculate the distance between anchor and positive, as well as the distance between anchor and negative, respectively:

where $\{\bm{W}_{d}\in\mathcal{R}^{d_{m}*d_{p}},\bm{b}_{d}\in\mathcal{R}^{d_{m}}\}$ are trainable parameters. $d_{m}$ is the hidden state size. $Dist(\cdot)$ is the distance calculation function, which we use Euclidean distance.

Since label semantic meaning is highly related to input text, we first leverage text representation $\bm{v}_{g}$ as the guidance to select necessary label representations from label embedding $\bm{E}$ as follows:

where $\{\bm{\omega}_{l}\in\mathcal{R}^{1*d_{a}},\bm{W}_{l}\in\mathcal{R}^{d_{a}*d_{p}},\bm{U}_{l}\in\mathcal{R}^{d_{a}*d_{p}}\}$ are trainable parameters. $d_{a}$ is the size of attention unit. $\bm{I}_{l}\in\mathbb{R}^{m}$ is a row vector of $1$. $\bm{U}_{l}\bm{v}_{g}\otimes\bm{I}_{l}$ means repeating $\bm{U}_{l}\bm{v}_{g}$ m times. $\bm{\beta}^{l}$ is the unnormalized attention weight for label representations. $\bar{\bm{h}}_{e}$ denotes the text supervised semantic vector generated from label semantics. With this operation, the unexpected noise from fine-grained descriptions can be effectively mitigated and label semantics can be expressed accurately.

Meanwhile, we intend to analyze the impact from label to text in a similar way. Since there are multiple labels, we adopt each label to guide the selection of input text. Then, max-pooling is employed to merge results for label supervised semantic vectors.

where $t\in\{1,2,...,m\}$ indicates the index of the $t^{th}$ label. $\beta^{t}$ is the unnormalized attention weight for word representations under the guidance of the $t^{th}$ label. $\{\bm{\omega}\in\mathcal{R}^{1*L_{s}},\bm{W}\in\mathcal{R}^{L_{s}*d_{p}},\bm{U}\in\mathcal{R}^{L_{s}*d_{p}}\}$ are also trainable parameters. $L_{s}$ is the sentence length. $\bar{\bm{h}}_{s}$ denotes the label supervised semantic vector from sentence semantics.

Table II reports the results on NLI task. We can conclude that our proposed R²-Net and DELE achieve highly comparable performance over all datasets. Moreover, DELE-RoBERTa-(base) has the best performance. To achieve such advantages, we first select PLMs as backbones, so that knowledge from large corpora can be accessed. This is one of the reasons that our methods outperform other baselines, especially better than these PLMs-free baselines (Table II(1)-(4)) by a large margin. Second, we develop a novel R² task to help models fully exploit label information in a one-hot manner. Along this line, R²-Net and DELE can obtain intra-class and inter-class knowledge among input texts with the same or different labels, which is in favor of achieving better performance than all baselines, including PLMs baselines (Table II(5)-(9)). Moreover, we design to incorporate fine-grained descriptions from WordNet into label embedding learning, so that particular label semantics can be better represented. This is another vital reason that DELE achieves the best performance, which also demonstrates the effectiveness of our proposed description-enhanced label embedding method.

Moreover, incorporating label information (one-hot encoding or label embedding) can better ensure model performance when dealing with difficult examples (Hard test in Table II), in which text sequences with obviously identical words have been removed [17]. By employing R² task to measure labels in a one-hot manner, R²-Net can better exploit implicit information from input text, which leads to an inspiring performance. Moreover, label embedding methods can exploit label information better than one-hot encoding methods. Therefore, we observe the superiority of label embedding methods. Furthermore, DELE employs fine-grained descriptions to enhance label embedding learning. Thus, it further improves the performance of label embedding methods and achieves the best performance.

To make a fair comparison, we replace basic encoders of label embedding methods (e.g., LEAM, EXAM) with the same PLMs backbones as our methods did, and report results in Table II(10)-(15), in which DELE still has better performance. The advantages of DELE lie in the more advanced label encoder and the consideration of denoising operation for additional knowledge. For label encoder, DELE not only learns the embedding during the model training so that the learned results are suitable for the current task, but also employs multi-aspect real sentences from WordNet as descriptions to enhance label semantic modelling. Meanwhile, considering that introducing fine-grained descriptions will import unexpected noise, we develop a novel mutual interaction module based on CL to leverage attention mechanism to select the most relevant parts from input text and labels simultaneously. Then, with the help of CL framework, DELE is able to use proper fine-grained knowledge to enhance the understanding of labels, and improve the model performance.

Apart from NLI task, we also select PI task to evaluate the model performance. PI task concerns whether two sentences express the same meaning and has broad applications in question answering communities⁴⁴4https://www.quora.com/. Table IV reports corresponding results. We observe that R²-Net and DELE still achieve highly competitive performance over other baselines. For one thing, the results demonstrate that though PI task is a binary classification task, its labels still contain some useful semantics. For another, we can conclude that our proposed R² task and label embedding methods with fine-grained descriptions are useful for exploiting label information and boosting model performance.

Besides, we obtain that almost all models have a better performance on Quora than MSRP. One possible reason is that Quora has more data (over 400k sentence pairs in Quora v.s. 5,801 sentence pairs in MSRP). Besides, inter-sentence interaction is probably another reason. Lan et al. [66] has observed that Quora dataset contains many sentence pairs with less complicated interactions (many identical words in sentence pairs). Therefore, we can obtain that ALBERT achieves the best performance on MSRP dataset.

To make a better evaluate, we further conduct experiments on SST-5 and Yahoo! Answer datasets, which have more complex and realistic labels (e.g., somewhat positive, Business). Table IV summarizes the results. Similarly, our proposed methods achieve stable and comparable performance. Moreover, Different from previous experiments, label embedding baselines (i.e., EXAM, LGDSC) do not achieve a better improvement, compared with PLMs baselines. One possible reason is that embedding these complicated labels with coarse-grained information cannot capture sufficient useful information for label exploitation and text classification.

Moreover, compared with PLMs baselines, the improvement of DELE is not so obvious. After analyzing the dataset deeply, we observe that some QA pairs in this dataset have more than 512 words, as well as irregular textual expressions. These input noises will do harm to label utilization since we leverage text representations to guide the denoising process of label description utilization. Meanwhile, we only select no more than 3 descriptions manually, which may be insufficient for those labels that have a wealth of semantics. Furthermore, the particular meaning of label words requires a precise selection of label descriptions. However, DELE leverages attention mechanism to select the most relevant parts from label embedding $\bm{E}$, where vanilla embedding and description embedding have already been integrated. Applying denoising operation at an earlier stage may achieve better performance. Nevertheless, on the contrary, DELE has made an early attempt at leveraging fine-grained knowledge to enhance label embedding and denoising of knowledge utilization, which also illustrates the advancement of DELE.

As illustrated in Eq. 4 and Eq. 20, kernel sizes $d_{k}$ in R²-Net, and $\{\delta,\mu\}$ in DELE are essential for model performance. To this end, we conduct additional parameter sensitive test to verify their impact.

For kernel sizes $d_{k}$, it aims to extract local information of input text at different scales. Results with different kernel size settings are reported in Fig. 5(A). We can observe that R²-Net will have better performance with more kernel sizes, in which $\{1,2,3\}$, $\{1,2,5\}$, and $\{2,3,5\}$ achieve the best performance. Moreover, using $\{1,2,3\}$ as kernel sizes have more stable performance. We speculate the possible reason is that these kernel sizes can pay more attention to uni-gram, bi-gram, as well as tri-gram information, which is helpful for enhancing the representation learning of PLMs. Meanwhile, using $\{2,3,5\}$ as kernel sizes has similar performance, which also demonstrates its effectiveness.

For $\sigma$ and $\mu$, when removing CL or R² task (i.e., $\delta=0$ or $\mu=0$), model performance cannot be comparable with the performance with CL or R² task (i.e., $\delta\neq 0$ or $\mu\neq 0$), demonstrating the necessity of these two SSL frameworks. Moreover, with the increasing of label size or label semantics, the best values of $\delta$ and $\mu$ also become larger. For one thing, this phenomenon proves that our proposed SSL can extract more critical information from more diverse and realistic labels, and have a bigger impact on model performance, proving the necessity of better label utilization methods. Furthermore, we observe that $\mu$ has a bigger impact than $\alpha$ when labels are harder, which is somewhat counterintuitive and different from other results. The possible reason is the underutilization of label information. Limited information and unexpected noise will decrease the effectiveness of our proposed CL framework. This finding is similar to the observation in Section VI-D3.

The overall experiments have proved the superiority of our proposed methods. However, which parts in R²-Net and DELE play a more important role in label utilization and performance improvement is still unclear. Therefore, we perform an ablation study to conduct a comprehensive analysis. Corresponding results are reported in Table V and Table VI. Note that we select BERT-(base) as the backbone to compare the importance of each part. Leveraging RoBERTa-(base) as the backbone will obtain similar results.

(a) For R²-Net, we conduct experiments on CNN-based local encoder, R² task classification, as well as loss function. Results are summarized in Table V. According to the results, we can observe varying degrees of model performance decline. Among all of them, R² task has the biggest impact, and triple loss has a relatively small impact on the model performance. Replacing triplet loss with contrastive loss will improve model performance slightly. These observations prove that R² task is more important for relation information utilization. R² task and better contrastive loss will further improve model performance.

(b) For DELE, we focus on label embedding module, mutual interaction module, and R² task. Results are reported in Table VI. Here, “BERT+Atten” denotes leveraging vanilla label embedding to select important parts of text without CL “$\bar{\bm{h}}_{s}$ (w/o mutual interaction)” indicates leveraging $\bar{\bm{h}}_{s}$ for classification and no interactions between input text and labels. Other ablation experiments have similar settings. According to the results, we obtain some observations. First of all, results (3-4) demonstrate the necessity of mutual interactions between input text and labels, especially result (4). When there is no interaction between input text and labels, label semantics cannot handle final classification at all. Second, results (5-6) prove that utilizing fine-grained descriptions is helpful for semantic understanding and performance improvement. Moreover, results (7-8) inspire us that R² task can provide additional signals for label utilization, which demonstrates the effectiveness and necessity of better label utilization methods. Last but not least, the comparison between $\bar{\bm{h}}_{s}$ and $\bar{\bm{h}}_{e}$ with the same setting (e.g., w/o description) indicates that CL can constrain learned representations from different views to have similar semantics. However, $\bar{\bm{h}}_{e}$ cannot be the replacement for input text, only an enhancement to input text.

To provide some intuitionistic examples for explaining the superiority of our models, we sample 1,500 sentence pairs from SNLI Test and send them to BERT-(base) baseline, EXAM-BERT-(base) baseline, R²-Net, and DELE to generate representations. Then, we use t-sne [67] to visualize them with the same parameter settings. Fig. 6 reports corresponding results. By comparing these four figures, we observe that representations generated by DELE have closer intra-class distances and more distinguishable inter-class distances. In other words, by considering fine-grained knowledge and mutual interactions, DELE is able to pull together samples with the same label and push apart samples with different labels, which is helpful for better text classification. These observations not only explain why DELE achieves impressive performance, but also demonstrate the necessity of more detailed and cleaner label descriptions, as well as more comprehensive analysis between input text and labels. All of these are very helpful for label semantics utilization and text classification performance improvement.