CIL: Contrastive Instance Learning Framework for Distantly Supervised Relation Extraction

BERT Encoder (Transformer Blocks, see Figure 2) transforms the above embedding inputs (token embedding & position embedding) into hidden feature vectors: ($h_{1},h_{2},\dots h_{e_{1}}\dots h_{e_{2}}\dots h_{L})$, where $h_{e_{1}}$ and $h_{e_{2}}$ are the feature vectors corresponding to the entities $e_{1}$ and $e_{2}$. By concatenating the two entity hidden vectors, we can obtain the entity-aware sentence representation $h=[h_{e_{1}};h_{e_{2}}]$ for the input sequence $x$. We denote the sentence encoder $\mathcal{H}$ as:

$$\mathcal{H}(x)=[h_{e_{1}};h_{e_{2}}]=h$$

Under the MIL framework, a couple of instances $x$ with the same relational triple $[e_{1},e_{2},r]$ form a bag $B$. We aim to design a bag encoder $\mathcal{F}$ to obtain representation $\widetilde{B}$ for bag $B$, and the obtained bag representation is also a representative of the current relational triple $[e_{1},e_{2},r]$, which is defined as:

$$\mathcal{F}(B)=\mathcal{F}([e_{1},e_{2},r])=\widetilde{B}$$

With the help of the sentence encoder described in section 2.2, each instance $x_{i}$ in bag $B$ can be first encoded to its entity-aware sentence representation $h_{i}=\mathcal{H}(x_{i})$. Then the bag representation $\widetilde{B}$ can be regarded as an aggregation of all instances’ representations, which is further defined as:

$$\mathcal{F}([e_{1},e_{2},r])=\widetilde{B}=\sum_{i=1}^{K}\alpha_{i}h_{i}$$

where $K$ is the bag size. As for the choice of weight $\alpha_{i}$, we follow the soft attention mechanism used in (Lin et al., 2016), where $\alpha_{i}$ is the normalized attention score calculated by a query-based function $f_{i}$ that measures how well the sentence representation $h_{i}$ and the predict relation $r$ matches:

$$\alpha_{i}=\frac{e^{f_{i}}}{\sum_{j}e^{f_{j}}}$$

where $f_{i}=h_{i}Aq_{r}$, $A$ is a weighted diagonal matrix and $q_{r}$ is the query vector which indicates the representation of relation $r$ (randomly initialized).

Then, to train such a bag encoder parameterized by $\theta$, a simple fully-connected layer with activation function softmax is added to map the hidden feature vector $\widetilde{B}$ to a conditional probability distribution $p(r|\widetilde{B},\theta)$, and this can be defined as:

$$p(r|\widetilde{B},\theta)=\frac{e^{o_{r}}}{\sum_{i=1}^{n_{r}}e^{o_{i}}}$$

where $o=M\widetilde{B}+b$ is the score associated to all relation types, $n_{r}$ is the total number of relations, $M$ is a projection matrix, and $b$ is the bias term.

And we define the objective of bag encoder using cross-entropy function as follows:

$$\mathcal{L_{B}}(\theta)=-\sum_{i=1}\log p(r_{i}|\widetilde{B}_{i},\theta)$$

As illustrated in section 1, the goal of our framework CIL is that the instances containing the same relational triples (i.e.positive pairs) should be as close (i.e.$\sim$) as possible in the hidden semantic space, and the instances containing different relational triples (i.e.negative pairs) should be as far (i.e.$\nsim$) away as possible in the space. A formal description is as follows.

Assume there is a batch bag input (with a batch size $G$): $(B_{1},B_{2},\dots,B_{G})$, the relational triples of all bags are different from each other. Each bag $B$ in the batch is constructed by a certain relational triple $[e_{1},e_{2},r]$, and all instances $x$ inside the bag satisfy this triple. The representation of the triple can be obtained by bag encoder as $\widetilde{B}$.

We pick any two bags $B_{s}$ and $B_{t:t\neq s}$ in the batch to further illustrate the process of contrastive instance learning. $B_{s}$ is defined as the source bag constructed with relational triple $[e_{s1},e_{s2},r_{s}]$ while $B_{t}$ is the target bag constructed with triple $[e_{t1},e_{t2},r_{t}]$. And we discuss the positive pair instance and negative pair instances for any instance $x_{s}$ in bag $B_{s}$.

It is worth noting that all bags are constructed automatically by the distantly supervised method, which extracts relational triples from instances in a heuristic manner and may introduce true/false positive label noise to the generated data. In other words, though the instance $x$ is included in the bag with relational triple $[e_{1},e_{2},r]$, it may be noisy and fail to express the relation $r$.

As discussed above, for any instance $x_{s}$ in the source bag $B_{s}$: (1) The instance $x_{s}^{*}$ after controllable data augmentation based on $x_{s}$ is its positive pair instance. (2) The relational triple representations $\widetilde{B}_{t}$ of other different ($t\neq s$) bags in the batch are its negative pair instances. The overall schematic diagram of CIL is shown in Figure 9.

And we define the objective for instance $x_{s}$ in bag $B_{s}$ using InfoNCE (Oord et al., 2018) loss:

$$\mathcal{L_{C}}(x_{s};\theta)=-\log\frac{e^{\text{sim}(h_{s},h_{s}^{*})}}{e^{\text{sim}(h_{s},h_{s}^{*})}+\sum_{t:t\neq s}e^{\text{sim}(h_{s},\widetilde{B}_{t})}}$$

where $\text{sim}(a,b)$ is the function to measure the similarity between two representation vectors $a,b$, and $h_{s}=\mathcal{H}(x_{s}),h_{s}^{*}=\mathcal{H}(x_{s}^{*})$ are the sentence representations of instances $x_{s},x_{s}^{*}$.

Besides, to inherit the ability of language understanding from BERT and avoid catastrophic forgetting (McCloskey and Cohen, 1989), we also add the masked language modeling (MLM) objective to our framework. Pre-text task MLM randomly masks some tokens in the inputs and allows the model to predict the masked tokens, which prompts the model to capture rich semantic information in the contexts. And we denote this objective as $\mathcal{L_{M}}(\theta)$.

Accordingly, the total training objective of our contrastive instance learning framework is:

$$\mathcal{L}(\theta)=\frac{\lambda(t)}{N}\sum_{B}\sum_{x\in B}\mathcal{L_{C}}(x;\theta)+\mathcal{L_{B}}(\theta)+\mathcal{\lambda_{M}}\mathcal{L_{M}}(\theta)$$

where $N=KG$ is the total number of instances in the batch, $\mathcal{\lambda_{M}}$ is the weight of language model objective $\mathcal{L_{M}}$, and $\lambda(t)\subset[0,1]$ is an increasing function related to the relative training steps $t$:

$$\lambda(t)=\frac{2}{1+e^{-t}}-1$$

At the beginning of our training, the value of $\lambda(t)$ is relatively small, and our framework CIL focuses on obtaining an accurate bag encoder ($\mathcal{L_{B}}$). The value of $\lambda(t)$ gradually increases to 1 as the relative training steps $t$ increases, and more attention is paid to the contrastive instance learning ($\mathcal{L_{C}}$).

We evaluate our method on three popular DSRE benchmarks — NYT10, GDS and KBP, and the dataset statistics are listed in Table 1.

Following previous literature (Lin et al., 2016; Vashishth et al., 2018; Alt et al., 2019), we first conduct a held-out evaluation to measure model performances approximately on NYT10-D and GDS. Besides, we also conduct an evaluation on two human-annotated datasets (NYT10-H & KBP) to further support our claims. Specifically, Precision-Recall curves (PR-curve) are drawn to show the trade-off between model precision and recall, the Area Under Curve (AUC) metric is used to evaluate overall model performances, and the Precision at N (P@N) metric is also reported to consider the accuracy value for different cut-offs.

We choose six recent methods as baseline models.

We summarize the model performances of our method and above-mentioned baseline models in Table 2. From the results, we can observe that: (1) On both two datasets, our proposed framework CIL achieves the best performance in all metrics. (2) On NYT10-D, compared with the previous SOTA model DISTRE, CIL improves the metric AUC (42.2$\to$50.8) by 20.4% and the metric P@Mean (66.8$\to$86.0) by 28.7%. (3) On GDS, though the metric of previous models is already high ($\approx 90.0$), our model still improves it by nearly 2 percentage points. (89.9$\to$91.6 & 92.8$\to$94.1).

The overall PR-curve on NYT10-D is visualized in Figure 10. From the curve, we can observe that: (1) Compared to PR-curves of other baseline models, our method shifts up the curve a lot. (2) Previous SOTA model DISTRE performs worse than model RESIDE at the beginning of the curve and yields a better performance after a recall-level of approximately 0.25, and our method CIL surpasses previous two SOTA models in all ranges along the curve, and it is more balanced between precision and recall. (3) Furthermore, as a SOTA scheme of relation learning, MTB fails to achieve competitive results for DSRE. This is because MTB relies on label information for pre-training, and noisy labels in DSRE may influence its model performance.

The automated held-out evaluation may not reflect the actual performance of DSRE models, as it gives false positive/negative labels and incomplete KB information. Thus, to further support our claims, we also evaluate our method on two human-annotated datasets, and the results²²2Manual evaluation is performed for each test sentence. are listed in Table 3.

From the above result table, we can see that: (1) Our proposed framework CIL can still perform well under accurate human evaluation, with averagely 21.7% AUC improvement on NYT10-H and 36.2% on KBP, which means our method can generalize to real scenarios well. (2) On NYT10-H, DISTRE fails to surpass PCNN-ATT in metric P@Mean. This indicates that DISTRE gives a high recall but a low precision, but our method CIL can boost the model precision (54.1$\to$63.0) while continuously improving the model recall (37.8$\to$46.0). And the human evaluation results further confirm the observations in the held-out evaluation described above.

We also present the PR-curve on KBP in Figure 11. Under accurate sentence-level evaluation on KBP, the advantage of our model is more obvious with averagely 36.2% improvement on AUC, 17.3% on F1 and 3.9% on P@Mean, respectively.

To further understand our proposed framework CIL, we also conduct ablation studies.

We firstly conduct an ablation experiment to verify that CIL has utilized abundant instances inside bags: (1) By removing our proposed contrastive instance learning, the framework degenerates into vanilla MIL framework, and we train the MIL on regular bags (MIL_bag). (2) To prove the MIL can not make full use of sentences, we also train the MIL on sentence bags (MIL_sent), which repeats each sentence in the training corpus to form a bag³³3All the results in Table 4 are obtained under the same test setting that uses MIL bags (i.e.BERT+ATT) as test units..

From Table 4 we can see that: (1) MIL_bag only resorts to the accurate bag-level representations to train the model and fails to play the role of each instance inside bags; thus, it performs worse than our method CIL (50.8$\to$40.3). (2) Though MIL_sent can access all training sentences, it loses the advantages of noise reduction in MIL_bag (40.3$\to$30.6). The noisy label supervision may wrongly guide model training, and its model performance heavily suffers from DS data noise (86.0$\to$63.3). (3) Our framework CIL succeeds in leveraging abundant instances while retaining the ability to denoise.

To validate the rationality of our proposed positive/negative pair construction strategy, we also conduct an ablation study on three variants of our framework CIL. We denote these variants as:

And we summarize the model performances of our CIL and other three variants in Table 5.

As the previous analysis in section 2.4, the three variants of our CIL framework may suffer from DS noise: (1) Both variants CIL_randpos and CIL_bagpos may construct noisy positive pairs; therefore, their model performances have a little drop (50.8$\to$49.2, 50.8$\to$47.8). Besides, the variant CIL_bagpos also relies on the bag encoder, for which it performs worse than the variant CIL_randpos (49.2$\to$47.8). (2) Though the constructed negative pairs need not be as accurate as positive pairs, the variant CIL_randneg treats all instances equally, which gives up the advantage of formed accurate representations. Thus, its model performance also declines (50.8$\to$48.4).

We select a typical bag (see Table 6) from the training set to better illustrate the difference between MIL_sent, MIL_bag and our framework CIL.

Under MIL_sent pattern, both S1, S2 are used for model training, and the noisy sentence S2 may confuse the model. As for MIL_bag pattern, S1 is assigned with a high attention score while S2 has a low attention score. However, MIL_bag only relies on the bag-level representations, and sentences like S2 can not be used efficiently. Our framework CIL makes full use of all instances (S1, S2) and avoids the negative effect of DS data noise from S2.