The Short Text Matching Model Enhanced with Knowledge via Contrastive Learning

Given two original sentences $s_{1}$ and $s_{2}$, the objective of STM is to determine whether they have the same semantic meaning. It can be considered as a binary classification task. To capture more comprehensive semantic information of $s_{1}$ and $s_{2}$, we generated complement sentences $s^{{}^{\prime}}_{1}$ and $s^{{}^{\prime}}_{2}$ specifically for them. The details of the generation process can be found in Section 3.6.

Given the remarkable performance of the pre-trained model BERT[15] in a range of natural language processing tasks, we utilize it as the encoder to separately encode the original and complementary sentences, generating high-dimensional vectors. Specifically, we first insert the $[CLS]$ token to the beginning of each sentence, and then encode them using BERT. Finally, by extracting the representation vector of the $[CLS]$ token, we obtain the sentence representations $h_{1}\in\mathbb{R}^{1\times d}$ and $h_{2}\in\mathbb{R}^{1\times d}$ for the original sentences and $h^{{}^{\prime}}_{1}\in\mathbb{R}^{1\times d}$ and $h^{{}^{\prime}}_{2}\in\mathbb{R}^{1\times d}$ for the complementary sentences. $d$ is the embedded dimension. It is worth noting that reduce the parameters count and prevent over-fitting, each sentence shares the same encoder.

Finally, we combine the semantic representations of the complementary sentence with those of the original sentence to obtain an enhanced sentence representation $h_{e1}$ and $h_{e2}$. This process improves the model’s ability to discern the semantic similarity between two sentences.

$$h_{e1}=[h_{1};h^{\prime}_{1}]$$

(1)

$$h_{e2}=[h_{2};h^{\prime}_{2}]$$

(2)

The keywords of a sentence contain crucial information that the sentence intends to convey. By analyzing the relationships between keywords, the model can better infer the semantic relationship between two sentences. Therefore, to extract and represent the relationships between keywords, In this module, we utilize the keywords from the original sentence to retrieve relevant knowledge words from external knowledge sources[4]. Subsequently, we construct a graph with these keywords and knowledge words as nodes, and the edges of the graph represent the relationships between words. Finally, we utilize the GCN[16] to extract the relationship features between words and obtain a graph representation vector.

During the prediction process, our model combines the representations of sentences and graphs to predict the similarity scores between the original sentences.

$$h_{final}=[h_{e1};h_{e2};|h_{e1}-h_{e2}|;h_{graph}]$$

(6)

$$p=Sigmode(w^{T}h_{final}+b)$$

(7)

To facilitate the capturing of shared features and the differentiation of non-shared features between original and complement text, we propose a semi-supervised contrastive learning strategy[18] for integrating the enhanced layers. We consider the encoded vectors of original and corresponding complement sentences as positive examples, since they are semantically very close. Moreover, to obtain more negative examples, we regard other complement sentences in the same batch as negative examples for the original sentence. By optimizing the contrastive learning, the model will reduce the distance between the encoded vectors of original and corresponding complement sentences while enlarging the distance between the encoded vectors of original and other unrelated sentences. As a result, the final model can better differentiate the shared semantic parts between original and complement sentences.

To be more specific, firstly, we obtain the encoded vectors of the original sentences and the complement sentences for conducting contrastive learning, respectively.

$$\hat{h}_{1}=FNN(h_{1})$$

(8)

$$\hat{h}_{2}=FNN(h_{2})$$

(9)

$$\hat{h}^{{}^{\prime}}_{1}=FNN(h^{\prime}_{1})$$

(10)

$$\hat{h}^{{}^{\prime}}_{2}=FNN(h^{\prime}_{2})$$

(11)

The training objective is to minimize the semantic distance between the original sentence and the complement sentence, while maximizing the distance between unrelated sentences. We utilize InfoNCE loss[19] as the training loss and cosine similarity to measure the semantic distance between sentences.

$$\mathcal{L}_{\text{contrast1}}=-\sum_{i=1}^{N}\log\frac{e^{\operatorname{sim}\left(\hat{h}_{i,1},\hat{h}_{i,1}^{\prime}\right)/\tau}}{\sum_{j=1}^{BatchSize}e^{\operatorname{sim}\left(\hat{h}_{i,1},h_{j,1}^{{}^{\prime}}\right)/\tau}}$$

(12)

$$\mathcal{L}_{\text{contrast2}}=-\sum_{i=1}^{N}\log\frac{e^{\operatorname{sim}\left(\hat{h}_{i,2},\hat{h}_{i,2}^{\prime}\right)/\tau}}{\sum_{j=1}^{BatchSize}e^{\operatorname{sim}\left(\hat{h}_{i,2},h_{j,2}^{{}^{\prime}}\right)/\tau}}$$

(13)

where $N$ is the total number of samples. $h_{j,k}^{{}^{\prime}}$ represents the $kth$ original sentence of the $jth$ sample in the current batch. $\tau$ is the temperature hyperparameter, which is used to control the model’s differentiation of negative samples.

We apply the BCE loss to optimize the sentence matching task during the training process.

$$\mathcal{L}_{\text{binary}}=-\sum_{i=1}^{N}y_{i}log(p_{i})+(1-y_{i})log(1-p_{i})$$

(14)

The final total training loss function:

$$\mathcal{L}=\alpha\mathcal{L}_{\text{binary}}+\beta\mathcal{L}_{\text{contrast1}}+\gamma\mathcal{L}_{\text{contrast2}}$$

(15)

$\alpha$, $\beta$, and $\gamma$ are hyperparameters set to 0.8, 0.1, 0.1 respectively.

To generate high-quality complement sentences, we constructed a dataset for training the SimBERT model. We can generate complementary sentences for existing data through this model. The model improves the attention mask mechanism in Transformer[20], resulting in efficient performance when generating similar sentences.

Specifically, during the data collection stage, we collected user query statements and corresponding advertisements returned by search engines (UC and Quark). For advertisements with low click-through rates, we considered them irrelevant to the user’s search intent and query statement. Therefore, we filtered out ad texts below the threshold $K$, and paired ad texts above the threshold with their corresponding queries to form sentence pairs. During the training phase, we obtained a total of 30,000k sentence pairs for model training. Finally, in the testing phase, the trained SimBERT model was frozen and used to generate complement sentences for the original sentences in the public dataset.

We applied Bert-base-chinese as our encoder and AdamW as the optimizer during the training phase. In addition, we implemented a hierarchical learning rate strategy, with a learning rate of 1e-4 for the pre-training model and 2e-5 for the other parts. The model was trained for 30 epochs on a GPU server with a 3080 GPU, using a batch size of 16. Early stopping was employed.

To validate the effectiveness of the proposed KSTM model, we conducted experiments on two publicly available Chinese datasets, i.e.,BQ[21] and LCQMC[22].

Bank Question(BQ): This is a large-scale question matching dataset related to the banking and finance domain, which has been widely used in sentence matching tasks. It contains a total of 120k sentence pairs, with the training, validation, and test sets containing 100k, 10k, and 10k sentence pairs, respectively.

Large-scale Chinese Question Matching Corpus (LCQMC): This is a semantic matching dataset of questions collected from the Baidu Knowledge website. The dataset focuses on matching intentions rather than meanings. It contains a total of 260k sentence pairs, with the training, validation, and test sets containing 239k, 8.4k, and 12.5k sentence pairs, respectively.

Compared to previous research, we evaluated our approach on two publicly available datasets using the same evaluation metrics as previous studies, i.e., $ACC$ and $F1$. The calculation formula of $ACC$ and $F1$ are as follows,

$$ACC=\frac{TP+TN}{TP+TN+FP+FN}$$

(16)

$$P=\frac{TP}{TP+FP}$$

(17)

$$R=\frac{TP}{TP+FN}$$

(18)

$$F1=\frac{2*P*R}{P+R}$$

(19)

where $TP$ and $FP$ respectively represent the number of cases correctly predicted as positive and the number of cases incorrectly predicted as positive. $TN$ and $FN$ respectively represent the number of cases correctly predicted as negative and the number of cases incorrectly predicted as negative.

To demonstrate the effectiveness of our approach, we compared KSTM model with the following strong baseline models, which use BERT as encoder.

Glyce[23]: A glyph-based model for Chinese NLP tasks.

GMN[8]: A sentence matching framework capable of dealing with multi-granular input information.

LET[2]: A short Text Matching model using external knowledge to eliminate sentence ambiguity.

DSSTM[9]: Combining the relative distance information of words, a distance-aware self-attention and multi-level matching model is proposed to solve the Chinese short Text Matching task.

OTE[3]: A short Text Matching model based on external knowledge, combined with SoftLexico model and hybrid data augmentation to obtain multi-granularity features.

CBM[1]: This method proposes to use external knowledge to enhance the semantic information of the original short text.

We compared our KSTM model with other short Text Matching models that use pre-trained Bert models as encoders on two datasets, using $F1$ and $accuracy$ as evaluation metrics. As shown in Table 1, the KSTM model outperforms all baseline models on both datasets. On the BQ dataset, the KSTM model achieves 1.46% higher accuracy and 1% higher $F1$ than the present SOTA model CBM. On the LCQMC dataset, the KSTM model outperforms the present best model DSSTM by 0.1% in accuracy and 1.1% in $F1$. This demonstrates the effectiveness of our proposed KSTM model.

In order to demonstrate the effectiveness of the components of KSTM, we conducted three ablation experiments on the BQ and LCQMC datasets, as shown in Table 2. The first experiment aimed to verify the effectiveness of contrastive learning. We removed contrastive learning between the original sentence and its complementary sentence and used only binary cross-entropy as the overall loss function. The second experiment aimed to demonstrate the improvement in model performance by introducing external knowledge into the model. We did not use external knowledge graphs or graph encoding information, only enhanced sentence encoding for sentence matching. The third experiment aimed to demonstrate the effectiveness of the complementary sentences we constructed. Following the unsupervised contrastive learning approach proposed by SimCES[24], we replaced the complementary sentences with the original sentences, we inputted the original sentences into the model to obtain the encodings, which were not identical due to the presence of dropout. Two identical original sentences were used as positive examples for contrastive learning.

The experimental results demonstrate that removing any part of the model will have a negative impact on its performance. Specifically, when we remove contrastive learning, the model’s accuracy and F1 score on the BQ dataset decrease by 1.52% and 0.94%, respectively. On the FCQMC dataset, the accuracy and F1 score decrease by 0.9% and 1.4%, respectively. When we remove the knowledge graph, the model’s accuracy and F1 score on the BQ dataset decrease by 2.32% and 2.56%, respectively. On the FCQMC dataset, the accuracy and F1 score decrease by 3.62% and 3.35%, respectively. When we remove the use of complementary sentences and use unsupervised contrastive learning instead, the model’s accuracy and F1 score on the BQ dataset decrease by 1.94% and 2.94%, respectively. On the FCQMC dataset, the accuracy and F1 score decrease by 4.7% and 4.1%, respectively.

Through comparing three ablation strategies, we found that compared to external knowledge, removing contrastive learning has a smaller impact on the model. We believe this is because the role of contrastive learning is to guide the model to generate more semantically meaningful encoding. It can guide the model to use semantic vector representations to represent text with similar semantics using vectors that are close in space, and use vectors that are far apart in space for text with different semantics. For short-text datasets, pre-trained models already have strong encoding capabilities to understand and encode their semantic information, so the effect of using contrastive learning to improve model performance is lower. In addition, the use of complementary sentences and knowledge graphs has a significant positive effect on model performance. This is because they both introduce external knowledge, which largely solves the problem of the model lacking sufficient information to judge whether two sentences match in short-text scenarios. Comparing the results of the two datasets, removing knowledge graphs or complementary sentences results in a greater performance decrease on the LCQMC dataset than on the BQ dataset. We believe this is because the sentence matching in the LCQMC dataset is more focused on sentence intent rather than meaning, which requires the model to have more information to understand the sentence and perform intent inference. Therefore, the method of using external knowledge has a more significant improvement effect on models trained on this dataset.