Few-shot Open Relation Extraction with Gaussian Prototype and Adaptive Margin

Relation extraction (RE) is a fundamental task in natural language processing (NLP) that aims to identify and classify semantic relationships between entities within a text. Existing researches perform supervised deep learning techniques to gain relations based on large-scale labeled data [9, 10, 11], which have shown significant improvements in the accuracy. However, data annotation is time-consuming and labor-intensive. Some researchers [12, 13, 14] use distant supervision methods to align text with external knowledge bases to obtain labels. Liang et al. [14] introduce a constraint graph to model the dependencies between labels and shares information between different relation nodes to alleviate long-tail problem. But automatically generated labels may contain noise and depend on the quality of external knowledge bases, affecting model training. With the development of few-shot learning, few-shot relation extraction has gradually become mainstream.

Providing only a small number of data samples, Han et al. [15] are the first to propose the concept of few-shot relation extraction (FsRE) and introduce the relation extraction dataset FewRel. Recent work on few-shot relation extraction focuses on three main directions, including meta-learning methods, especially prototype-based methods [16, 17], parameter optimization learning methods [3, 18, 19, 20] and large language model based methods [21]. Prototype-based methods aim to obtain the distribution of each category named prototype and assign test samples to the closest one. Xiao et al. [17] propose an adaptive hybrid mechanism based on typical prototype to integrate label information into the features of each category support instance. Parameter optimization learning methods use fine-tuning methods to optimize the parameters of the pre-trained language model to adapt to the RE task. Zhang et al. [18] take relationship descriptions as prompt inputs and randomly discards some to simulate the scenario where support set labels are not visible. Zhang et al. [20] utilize prompt-tuning to fine-tune PLM to integrate relationship information and original prototypes. Large language model based methods leverage the vast knowledge and contextual understanding capability to effectively perform tasks in few-shot scenarios. Zhang et al. [21] utilize an instruction alignment method to fine-tune LLMs and aligns the RE task to the QA task to enhance relation extraction performance. Our work focuses on prototype-based methods and hopes to stimulate the prototype’s ability to identify unknown classes, which is called “none of the above” (NOTA).

When NOTA is introduced into relation extraction task, an additional challenge arises: accurately distinguishing known classes while precisely identifying and separating the unknown classes. On image classification tasks, there are some studies on FsRE with NOTA. Che et al. [22] treat background class as a pseudo label to train the boundary between known classes and unknown classes. But for text classification tasks, there are much less related studies. BERT-PAIR [3] is the first method for this task, which pairs support and query samples for both known and NOTA classes based on a sequence model to calculate the similarity score. Besides, Li et al. [19] obtain pseudo labels for pre-training by extracting paraphrase and passes the universal knowledge to the tiny models.

In previous studies [4], prototypes were learned solely from original sentence information, with model performance constrained by biases from entities and contexts. We follow the Semi-Factual Representation (SFR) strategy proposed in our previous work [23], which learns representations through multiple debiased views to mitigate this limitation. As shown in Fig.2(a), the details are as follows.

Main View. This view marks the head and tail entity in the raw sentence $\bm{s}=(s_{1},s_{2},...,s_{L})$ with tokens $\langle{h}\rangle$, $\langle{/h}\rangle$ and $\langle{t}\rangle$, $\langle{/t}\rangle$, respectively. We denote the sentence after applying this view as $\bm{s^{m}}=(s_{1}^{m},s_{2}^{m},...,s_{L}^{m})$.

Head and Tail Debiased Views. These two views replace the head entity $h_{i}$ and the tail entity $t_{i}$ with their attribute features. For example, the head entity “Tennen Mountains” is replaced by its attribute “[Mountain]”. The sentences in head debiased view and tail debiased view are denoted by $\bm{s^{h}}=(s_{1}^{h},s_{2}^{h},...,s_{L}^{h})$ and $\bm{s^{t}}=(s_{1}^{t},s_{2}^{t},...,s_{L}^{t})$, respectively.

Context Debiased View. This view utilizes WordNet [24] to generate synonyms and randomly replaces 5% of the words in $\bm{s}$. The resulting sentence is denoted by $\bm{s^{c}}=(s_{1}^{c},s_{2}^{c},...,s_{L}^{c})$.

We then use BERT [25] to encode these four views $\bm{s^{j}}(j=m,h,t,c)$, obtaining feature representations $\bm{z^{m}}$, $\bm{z^{h}}$, $\bm{z^{t}}$, and $\bm{z^{c}}$, respectively.

The conventional prototype method only calculates the prototype anchor and radius, that is, the prototype is only regarded as a sphere with a certain radius in high-dimensional space [7]. It only considers the mean of samples, but ignores the variance information that accurately reflects the distribution. This is prone to inaccurate description, especially for samples near the decision boundary. To alleviate this problem, we assume that the relation $r$ follows a mixed Gaussian distribution aggregated from the features of four views and propose a Gaussian Mixture Module (GMM)-based strategy detailed as follows.

First, prompt templates are constructed for four views and inserted into the input embedding sequence as prefix tokens to do prompt-tuning for $K$ shots in a meta-task,

where $prompt^{j}(j=m,h,t,c)$ represents the prompt templates corresponding to various views. Different from traditional high-dimensional space distance estimation strategies, Mahalanobis distance is used instead of Eucilidean distance to better adapt the characteristics of Gaussian distribution, utilizing both the mean and variance information. Next, the mean vector $\bm{\mu}$ and diagonal variance matrix $\text{diag}(\bm{\mathit{v}})$ of the Gaussian space are calculated corresponding to the relation type $r$ and the following formulas are used to get the $\bm{\mu}$, $\bm{v}$ values of the main view and three debiased views in turn:

where $\theta$ is a learnable parameter which is the same for all views. The four views reflect the features of relation $r$ from different aspects, therefore, the overall feature of $r$ can be obtained by fusing them according to their respective weights. The following function is utilized to calculate the adaptive mixed Gaussian weights $w\in R^{4}$ of the four views:

where $w$ is a four-dimensional vector that reflects the relative weights of the four views which differs for different samples, and $\phi_{1}$ and $\phi_{2}$ are learnable parameters. Moreover, for any instance $x$ in $Q^{k}$, $Q^{u}$, the distance between it and the relation $r$ in the mixed Gaussian prototype space can be measured as:

where for $j=m,h,t,c$:

For the support set $S$, we apply Eq.(4) and take the average of positive instances to obtain the distance between all samples and candidate relations in the mixed Gaussian space to obtain the distribution feature of each relation category $r_{c}$.

Since in few-shot scenarios, the distribution of positive samples for a certain category has a key influence on the relation, we introduce the prototype range indicator $R_{c}$ to show the range of each category with positive samples. However, using only a single indicator $R_{c}$ to judge the category may cause the problem of sample misjudgment near the decision boundary between known and NOTA classes. In order to alleviate this problem, the adaptive margin of NOTA, $M_{c}$, is introduced. $M_{c}$ is another indicator that affects the boundary between positive and negative examples, reflecting the tolerance of the learned prototype to negative examples. We utilize the distances from the positive instances to the relation anchor $r_{c}$ to obtain $R_{c}$ and the distances from the negative instances to $r_{c}$ to obtain $M_{c}$ as follows:

where $h(.)$ is the quantile function which follows the principle that most positive instances should be within the prototype range $R_{c}$ and most negative instances should be outside $R_{c}+M_{c}$, $\tau_{1}$ is a learnable parameter that controls the boundary range, $\tau_{2}$ is a learnable parameter that controls the tolerance for negative instances, and $x^{+}_{i}$ and $x^{-}_{i}$ represent positive and negative instances respectively.

For the query set $Q=Q^{k}\cup Q^{u}$, we utilize Eq.(4) to do the same thing and obtain the distances between instances and each prototype anchor. To determine whether a instance $x$ belongs to relation $r_{c}$, the classification rules are as follows:

Besides, if instance $x$ meets the criteria of multiple classes, the instance with the smallest GMM distance $r_{c}$ will be selected, and if instance $x$ does not belong to any of the relations in the known set, it will belong to the NOTA class. In this way we can accurately predict the labels of the instances in the query set whether they belong to a known class or NOTA class.

In the training procedure, we find that one of the important reasons for the poor performance of prototype learning methods when facing the NOTA problem is that the boundary of NOTA is complex and difficult to accurately describe with a small number of samples. Therefore, we refer to the method of Song [22] to expand some negative instances outside the $R_{c}+M_{c}$ region in the background regions by a certain proportion, and these examples are called pseudo negative samples (PNS), which don’t belong to any of the classes and serve as negative samples of all classes. According to the research of Wang [26] et al., since negative instances close to the boundary will have a greater impact on the classification between known and unknown classes, a higher generation probability is assigned to these negative instances. For each meta-task, several points are sampled outside the range of $R_{c}+M_{c}$ in the feature vector space as pseudo negative example sets. Then, the probability of selecting a sample in the pseudo-negative example set is calculated by the ratio of its distance from the margin boundary to the closed distribution range according to the following formula:

where $p$ is a pseudo negative sample generated by this strategy, and is added to the support set $S$ of all classes as a negative instance. After adding pseudo negative samples, we update the value of the range $R_{c}$ and margin $M_{c}$ again by Eqs.(6) and (7).

Then, the contrastive learning strategy is used to optimize our model, and GPAM loss can be formulated as:

where $\lambda$ is a hyperparameter that controls the range of the prototype, and $\alpha$ and $\beta$ are adjustable hyperparameters in contrastive learning.

Our loss function can be divided into three components. The initial component focuses on minimizing the range of known classes $R_{c}$. The middle component ensures that positive examples are positioned as close as possible to the anchors of the known classes. The final component ensures that negative examples are not only distanced from the anchors of the known classes but also placed outside the boundary of $R_{c}+M_{c}$. Finally, we summarize our training and optimization process of GPAM in Algorithm 1.

Datasets. We perform experiments on the public relation extraction dataset: FewRel [3]. FewRel is a benchmark dataset designed for evaluating models in relation classification tasks. It features 100 diverse relation types with annotated examples sourced from Wikipedia, and contains 700 instances for each relation. We use meta-learning methods to randomly extract samples from the FewRel dataset according to different task settings to form meta-task datasets without redundant instances. Different from conventional few-shot learning task, we add NOTA instances to the meta-task datasets according to a certain NOTA rate.

Comparing Methods. We compare our model GPAM with the following outstanding baselines. These methods can be categorized into three groups. 1) FsRE models with NOTA. Proto-BERT [4]: the original prototype network algorithm; BERT-PAIR [3]: an approach to measure the similarity of sentence pairs; MCMN [8]: an approach using triplet paraphrase and meta-learning paradigm to do low-shot RE. 2) Those without NOTA. Proto-HATT [27]: a hybrid attention-based prototypical network; MLMAN [28]: a multi-level matching and aggregation prototypical network; REGRAB [29]: a Bayesian meta-learning approach to learn the posterior distribution of the prototype and solve the uncertainty of the prototype vector; CTEG [30]: a model to decouple high co-occurrence relations; HCRP [31]: an approach to introduce relation label information and distinguish task difficulty; SimpleFSRE [16]: a direct addition approach that fuses the embedding of relation description to the prototype representation; SaCon [32]: a framework using diverse viewpoints through instance-label pairs to capture intrinsic textual semantics. 3) Large language models. GPT-4o: an outstanding closed-source large language model developed by OpenAI; GLM-4: an open-source large language model developed by the Tsinghua Zhipu AI team.

Training Details. For GMM-prototype metric learning module, the length of prompt template is set to 100; For decision boundary learning module, the initial values of $\lambda$, $\tau_{1}$ and $\tau_{2}$ are set to 0.001, 0.1 and 0.2 respectively, and the ratio of pseudo negative sampling is set to 0.2; In the loss function, the positive impact parameter $\alpha$ is set to 1 and the negative impact parameter $\beta$ is set to 3 to ensure that negative instances have a greater influence in the contrastive learning process. For the training process, we choose the SGD algorithm with learning rate 0.0002, weight decay 0.0001 as the optimizer.

Results on FsRE with NOTA task. Results on FewRel dataset with NOTA are shown in Table 1 and total, known and NOTA are evaluated individually. The observations are as follows:

• 1.

  Compared with previous methods, our GPAM clearly achieves state-of-the-art performance on all settings. The total accuracy of our GPAM exceeds the previous best conventional model MCMN, improving by 5.11%, 4.15%, 8.19%, and 7.13% on four tasks respectively. Benefit from the three designed modules, GPAM achieved good results on the FsRE with NOTA task.

• 2.

  GPAM’s performance for NOTA class is particularly outstanding under the NOTA rate 0.5 setting. Compared with the previous best performing method MCMN, our GPAM improves the accuracy of NOTA class extraction by 8.85% $\sim$ 10.30% at NOTA rate 0.5. The reason is that GMM-based distance metric and adaptive margin are introduced, GPAM has achieved significant advancements in the classification of both known and unknown classes.

• 3.

  As the number of shots $K$ increases, the performance of GPAM becomes higher and more stable. When the number of shots increases from 1 to 5, the accuracy for the NOTA class increases from 84.25 to 93.25 at NOTA rate 0.15, and from 90.75 to 96.10 at NOTA rate 0.5. We can obtain better prototype of categories and more precise decision boundaries with more instances because the performance improvement is reasonable as the shot increases.

Moreover, to compare the performance changes as the NOTA rate improves, we follow the evaluation methods of BERT-PAIR [3], and all models are trained and tested under four different NOTA rates: 0%, 15%, 30%, 50%. The experimental results are shown in Fig.3. GPAM outperforms the compared methods across all NOTA rate settings and maintains better stability. As the NOTA rate increases, the performance of traditional models such as Proto-BERT declines to varying degrees. For the current mainstream large models, GPT-4o’s performance drops dramatically after adding NOTA samples compared to non-NOTA. GLM-4 performs better, but its performance gradually decreases as the NOTA rate increases. It drops by 12.92% compared to non-NOTA when NOTA is 0.5. These compared models exhibit significantly weaker discrimination ability for the NOTA class compared to known classes.

Results on FsRE task. We also test the performance on traditional relation extraction task without NOTA shown in Table 2. It should be noted in advance that since the FewRel dataset test set on Codalab³³3https://codalab.org is no longer open to the public, the validation set is used for all tests. Our GPAM achieves performance slightly inferior to the SOTA model, despite considering the inclusion of unknown class judgments in the zero-NOTA scenario. This shows that our method focuses on tasks including NOTA, but also has acceptable effects on the conventional FsRE task.

In order to verify the role of each view and study the impact of each debiased view on the main view, we perform ablation studies on three debiased views. Results are shown in Table 3. We can make the observations as follows.

• 1.

  The debiased information of all views has a benefit for all task settings. It can be seen that removing any view will cause performance to degrade to varying degrees.

• 2.

  The semi-factual representation strategy has a greater improvement in performance in more difficult tasks. Comparing only the main view with the original model performance, we can find that when the nota rate is 0.5, the introduction of debiased views significantly improves the performance by 5.09% and 6.30% respectively.

• 3.

  The effect will be better when head and tail views are used together. It can be seen that deleting both the head and tail views at the same time is better than deleting either one. This is because introducing head or tail view alone may lead to incomplete or unbalanced information, and combining the both can get a more reliable representation.

To validate the effectiveness of the key strategies in the GMM-prototype metric learning module and to quantify their respective impacts, we construct four variants of GPAM as follows and evaluate their performance on the FewRel dataset shown in Table 3. The analysis of the results is as follows.

Analysis on Guassian Distance: the variant model that uses Euclidean distance instead of Mahalanobis distance as the metric in Eq.(5), that is, treats the prototype as a sphere. Mahalanobis distance shows a more significant improvement in the 5-shot scenario, with increases of 7.70% and 6.38%, respectively. This indicates that Mahalanobis distance based on Gaussian distribution is more advantageous than Euclidean distance for prototype construction in scenarios with multiple samples.

Analysis on Multi-Prompt: the variant model that utilizes the same prompt template instead of multi-prompt. We modify Eq.(2) and set the prompt templates $prompt^{j}$ of the four views to be the same. Multi-prompt strategy has a more significant effect when the NOTA rate is higher, with improvements of 8.80% and 7.17% respectively.

Analysis on Mixed Weights: the variant model that uses the averaging stategy to aggregate four views instead of mixed Gaussian weights. We remove the weight computation formula Eq.(3), set the weights of the four views to be the same, and then use Eq.(4) to obtain the aggregated prototype distance. Mixed weights strategy balances the differences in the importance of different views to certain type of samples, and the effect is more significant in the case of multiple shots.

Analysis on Self-Attention Mechanism: the variant model that removes the self-attention mechanism used in the prototype. We modify Eq.(3) by eliminating the self-attention operation, which results of the prototype relying solely on the raw concatenated features without further refinement. Self-Attention has a slight effect compared to other strategies.

Overall, all four strategies have improved the model to a certain extent, and the effects of Mahalanobis distance and multi-prompt are more significant.

As shown in Table 3, we also conduct an ablation study on decision boundary learning module, analyzing the impact of the presence or absence of key strategies as follows.

Analysis on Class Margin $M_{c}$. In order to study the impact of margin presence and dynamics on performance, we construct two variants of GPAM: 1) The variant model that utilizes only range indicator $R_{c}$ without margin $M_{c}$ to classify unknown classes. We achieve this by setting $M_{c}$ to zero and remove this item in the loss function Eq.(9). It can be seen that in the case of a relatively simple task 5-way-1-shot 0.15, the increase of the margin strategy is small, but in the other three complex tasks, the presence or absence of margin has a great impact, with an increase of more than 6%. This indicates that measuring the boundary solely with the prototype range is inaccurate, and the margin is crucial. 2) The variant model that utilizes fixed margin instead of adaptive margin which changes with negative instances of the prototype. We achieve this by changing $M_{c}$ in the margin computation formula Eq.(7) and the loss function Eq.(9) to a fixed value. Taking 5-way-1-shot 0.15 task as an example, using a fixed margin reduces the performance by 5.87%, which is even more significant than simply removing the margin. One possible reason is that when there are fewer shots, using the same $M_{c}$ for different categories will lead to inaccurate boundary ranges.

Analysis on Pseudo Negative Sampling (PNS). We construct a variant model with no pseudo negative samples in the train dataset. It can be seen that the PNS strategy shows a modest improvement of only 1.12% for the 5-way-1-shot task with the NOTA rate of 0.15, while it achieved over 3% improvement for the other three tasks. We analyze the reason for the results and infer that for the 5-way-1-shot 0.15 task, adding pseudo negative examples expands the boundary range of prototype. As a result, more NOTA classes are misclassified as known classes, even though the overall performance is improved. In order to obtain the optimal negative sampling rate, we conduct experiments and plot the performance of different negative sampling rates as shown in Figure 4. The performance is optimal in most cases when the negative sampling rate is 0.2, and the effect drops significantly when the value is 0.4 or higher.

In order to intuitively demonstrate the effect of GPAM, we conduct a case study and artificially constructs a difficult and confusing 5-way-1-shot 0.5 meta tasks. For test requirements, we choose support and query set instances from five certain known classes, and select NOTA instances for query set from the remaining classes. And we choose BERT-PAIR [3] as a baseline model to compare it with the original GPAM and its three variant models: 1) $\text{GPAM}^{\#1}$ is the variant model that removes the semi-factual representation module; 2) $\text{GPAM}^{\#2}$ is the variant model that uses Euclidean distance instead of Mahalanobis distance as the metric; 3) $\text{GPAM}^{\#3}$ is the variant model that removes the margin strategy and only uses range for boundary division. We show some of the instances and the corresponding extraction results in a batch during the validation process in Tabel 4. Next, we analyze the results of the case study in detail.

Overall, GPAM performs best among the ten cases, correctly judging 90% of the cases, followed by $\text{GPAM}^{\#3}$. For most known class query instances such as (1)-(3), all five models obtain correct results. For the query instance (4), sentence clearly implies that Chauncey Davis is the defensive end, which is a position played on team. But BERT-PAIR and $\text{GPAM}^{\#2}$ fail to get the correct answer and misjudge it as the relation located in body of water. For the query instance (5), which is considered one of the most challenging instances in this batch, Magas was the godfather of Serbian organized crime, where Magas, as the godfather, was the mastermind of organized crime, so the relation is main subject. Only GPAM and $\text{GPAM}^{\#2}$ are correct, the other three models assume that the relation between entities is not any of the five and get the result NOTA. Instances (4) and (5) show that both the semi-factual representation and Mahalanobis distance metric strategies contribute to solving the few-shot problem of known classes.

For the unknown class query instances in Table 4, BERT-PAIR, GPAM and the variant $\text{GPAM}^{\#3}$ work relatively well in distinguishing NOTA classes, while $\text{GPAM}^{\#1}$ and $\text{GPAM}^{\#2}$ may misclassify a NOTA class as a known class multiple times. This phenomenon occurs because some NOTA class relations have similar semantics to known class relations, such as P641 (sport team) and P431 (position played on team), P59 (constellation position) and P206 (located in body of water). We select the instance (6) where both are wrong for detailed analysis, the sentence describes that the road stretches from the Pitt River Bridge in the east to the Port Mann Bridge in the west and runs right next to the Fraser River. The correct relation NOTA (P177), expresses the relation that the bridge Port Mann Bridge spans the river Fraser River, instead of locating in body of water (P206) or other relationships. This shows that our complete GPAM is more accurate in identifying NOTA classes.