OpenIE6: Iterative Grid Labeling and Coordination Analysis for Open Information Extraction

Open Information Extraction (OpenIE) is an ontology-free information extraction paradigm that generates extractions of the form (subject; relation; object). Built on the principles of domain-independence and scalability Mausam (2016), OpenIE systems extract open relations and arguments from the sentence, which allow them to be used for a wide variety of downstream tasks like Question Answering Yan et al. (2018); Khot et al. (2017), Event Schema Induction Balasubramanian et al. (2013) and Fact Salience Ponza et al. (2018).

End-to-end neural systems for OpenIE have been found to be more accurate compared to their non-neural counterparts, which were built on manually defined rules over linguistic pipelines. The two most popular neural OpenIE paradigms are generation (Cui et al., 2018; Kolluru et al., 2020) and labeling (Stanovsky et al., 2018; Roy et al., 2019).

Generation systems generate extractions one word at a time. IMoJIE Kolluru et al. (2020) is a state-of-the-art OpenIE system that re-encodes the partial set of extractions output thus far when generating the next extraction. This captures dependencies among extractions, reducing the overall redundancy of the output set. However, this repeated re-encoding causes a significant reduction in speed, which limits use at Web scale.

On the other hand, labeling-based systems like RnnOIE Stanovsky et al. (2015) are much faster (150 sentences per second, compared to 3 sentences of IMoJIE) but relatively less accurate. They label each word in the sentence as either S (Subject), R (Relation), O (Object) or N (None) for each extraction. However, as the extractions are predicted independently, this does not model the inherent dependencies among the extractions.

We bridge this trade-off though our proposed OpenIE system that is both fast and accurate. It consists of an OpenIE extractor based on a novel iterative labeling-based architecture — Iterative Grid Labeling (IGL). Using this architecture, OpenIE is modeled as a 2-D grid labeling problem of size $(M,N)$ where $M$ is a pre-defined maximum number of extractions and $N$ is the sentence length, as shown in Figure 1. Each extraction corresponds to one row in the grid. Iterative assignment of labels in the grid helps IGL capture dependencies among extractions without the need for re-encoding, thus making it much faster than generation-based approaches.

While IGL gives high precision, we can further improve recall by incorporating (soft) global coverage constraints on this 2-D grid. We use constrained training Mehta et al. (2018) by adding a penalty term for all constraint violations. This encourages the model to satisfy these constraints during inference as well, leading to improved extraction quality, without affecting running time.

Furthermore, we observe that existing neural OpenIE models struggle in handling coordination structures, and do not split conjunctive extractions properly. In response, we first design a new coordination analyzer Ficler and Goldberg (2016b). It is built with the same IGL architecture, by interpreting each row in the 2-D grid as a coordination structure. This leads to a new state of the art on this task, with a 12.3 pts improvement in F1 over previous best reported result Teranishi et al. (2019), and a 1.8 pts gain in F1 over a strong BERT baseline.

We then combine the output of our coordination analyzer with our OpenIE extractor, resulting in a further increase in performance (Table 1). Our final OpenIE system — OpenIE6 — consists of IGL-based OpenIE extractor (trained with constraints) and IGL-based coordination analyzer. We evaluate OpenIE6 on four metrics from the literature and find that it exceeds in three of them by at least 4.0 pts in F1. We undertake manual evaluation to reaffirm the gains. In summary, this paper describes OpenIE6, which

• •

  is based on our novel IGL  architecture,

• •

  is trained with constraints to improve recall,

• •

  handles conjunctive sentences with our new state-of-art coordination analyzer, which is 12.3 pts better in F1, and

• •

  is 10$\times$ faster compared to current state of the art and improves F1 score by as much as 4.0 pts.

Banko et al. (2007) introduced the Open Information Extraction paradigm (OpenIE) and proposed TextRunner, the first model for the task. Following this, many statistical and rule-based systems have been developed Fader et al. (2011); Etzioni et al. (2011); Christensen et al. (2011); Mausam et al. (2012); Del Corro and Gemulla (2013); Angeli et al. (2015); Pal and Mausam (2016); Stanovsky et al. (2016); Saha et al. (2017); Gashteovski et al. (2017); Saha and Mausam (2018); Niklaus et al. (2018).

Recently, supervised neural models have been proposed, which are either trained on extractions bootstrapped from earlier non-neural systems Cui et al. (2018), or on SRL annotations adapted for OpenIE Stanovsky and Dagan (2016). These systems are primarily of three types, as follows.

Labeling-based systems like RnnOIE Stanovsky et al. (2018), and SenseOIE Roy et al. (2019) identify words that can be syntactic heads of relations, and, for each head word, perform a single labeling to get the extractions. Jiang et al. (2020) extend these to better calibrate confidences across sentences. Generation-based systems Cui et al. (2018); Sun et al. (2018) generate extractions sequentially using seq2seq models. IMoJIE Kolluru et al. (2020), the current state of art in OpenIE, uses a BERT-based encoder and an iterative decoder that re-encodes the extractions generated so far. This re-encoding captures dependencies between extractions, increasing overall performance, but also makes it 50x slower than RnnOIE. Recently, span-based models Jiang et al. (2020) have been proposed, e.g., SpanOIE Zhan and Zhao (2020), which uses a predicate module to first choose potential candidate relation spans, and for each relation span, classifies all possible spans of the sentence as subject or object.

Concurrent to our work Ro et al. (2020) proposed Multi²OIE, a sequence-labeling model for OpenIE, which first predicts all the relation arguments using BERT, and then predicts subject and object arguments associated with each relation using multi-head attention blocks. Their model cannot handle nominal relations and conjunctions in arguments, which can be extracted in our iterative labeling scheme.

The benchmarks also differ in their scoring functions along two dimensions: (1) computing similarity for each (gold, system) extraction pair, (2) defining a mapping between system and gold extractions using this similarity. OIE16 computes similarity by serializing the arguments into a sentence and finding the number of matching words. It maps each system extraction to one gold (one-to-one mapping) to compute both precision and recall. Wire57 uses the same one-to-one mapping but computes similarity at an argument level. CaRB uses one-to-one mapping for precision but maps multiple gold to the same system extraction (many-to-one mapping) for recall. Like Wire57, CaRB computes similarity at an argument level.

For detecting coordination boundaries, Ficler and Goldberg (2016a) re-annotate the Penn Tree Bank corpus with coordination-specific tags. Neural parsers trained on this data use similarity and replacability of conjuncts as features Ficler and Goldberg (2016b); Teranishi et al. (2017). The current state-of-the-art system (Teranishi et al., 2019) independently detects coordinator, begin, and end of conjuncts, and does joint inference using Cocke–Younger–Kasami (CYK) parsing over context-free grammar (CFG) rules. Our end-to-end model obtains better accuracy than this approach.

Bhutani et al. (2019) propose an OpenIE system to get extractions from question-answer pairs. Their decoder enforces vocabulary and structural constraints on the output both during training and inference. In contrast, our system uses constraints only during training.

Given a sentence with word tokens $\{w_{1},w_{2},\ldots,w_{N}\}$ the task of OpenIE is to output a set of extractions, say $\{E_{1},E_{2},\ldots,E_{M}\}$, where each extraction is of the form (subject; relation; object). For a labeling-based system, each word is labeled as S (Subject), R (Relation), O (Object), or N (None) for every extraction. We model this as a 2-D grid labeling problem of size $(M,N)$, where the words represent the columns and the extractions represent the rows (Figure 2). The output at position $(m,n)$ in the grid ($L_{m,n}$) represents the label assigned to the $n^{th}$ word in the $m^{th}$ extraction.

We propose a novel Iterative Grid Labeling (IGL) approach to label this grid, filling up one row after another iteratively. We refer to the OpenIE extractor trained using this approach as IGL-OIE.

IGL-OIE is based on a BERT encoder, which computes contextualized embeddings for each word. The input to the BERT encoder is $\{w_{1},$ $w_{2},\ldots,w_{N},\emph{[is]},\emph{[of]},\emph{[from]}\}$. The last three tokens (referred as $st_{i}$ in Figure 3) are appended because, sometimes, OpenIE is required to predict tokens that are not present in the input sentence.²²2‘is’, ‘of’ and ‘from’ are the most frequent such tokens. E.g., “US president Donald Trump gave a speech on Wednesday.” will have one of the extractions as (Donald Trump; [is] president [of]; US). The appended tokens make such extractions possible in a labeling framework.

The contextualized embeddings for each word or appended token are iteratively passed through a 2-layer transformer to get their IL embeddings at different levels, until a maximum level $M$, i.e. a word $w_{n}$ has a different contextual embedding $IL_{m,n}$ for every row (level) $m$. At every level $m$, each $IL_{m,n}$ is passed though a fully-connected labeling layer to get the labels for words at that level (Figure 3). Embeddings of the predicted labels are added to the IL embeddings before passing them to the next iteration. This, in principle, maintains the information of the extractions output so far, and hence can capture dependencies among labels of different extractions. For words that were broken into word-pieces by BERT, only the embedding of the first word-piece is retained for label prediction. We sum the cross-entropy loss between the predicted labels and the gold labels at every level to get the final loss, denoted by $J_{CE}$.

OpenIE systems typically assign a confidence value to an extraction. In IGL, at every level, the respective extraction is assigned a confidence value by adding the log probabilities of the predicted labels (S, R, and O), and normalizing this by the extraction length.

We believe that IGL architecture has value beyond OpenIE, and can be helpful in tasks where a set of labelings for a sentence is desired, especially when labelings have dependencies amongst them.³³3IGL is a generalization of Ju et al. (2018). Their model can only label spans which are subsets of one another. We showcase another application of IGL for the task of coordination analysis in Section 5.

Our preliminary experiments revealed that IGL-OIE has good precision, but misses out important extractions. In particular, we observed that the set of output extractions did not capture all the information from the sentence (Table 1). We formulate constraints over the 2-D grid of extractions (as shown in Figure 2) which act as an additional form of supervision to improve the coverage. We implement these as soft constraints, by imposing additional violation penalties in the loss function. This biases the model to learn to satisfy the constraints, without explicitly enforcing them at inference time.

To describe the constraints, we first define the notion of a head verb as all verbs except light verbs (do, be, is, has, etc.). We run a POS tagger on the input sentence, and find all head verbs in the sentence by removing all light verbs.⁴⁴4We used the light verbs listed by Jain and Mausam (2016). For example, for the sentence, “Obama gained popularity after Oprah endorsed him for the presidency”, the head verbs are gained and endorsed. In order to cover all valid extractions like (Obama; gained; popularity) and (Oprah; endorsed him for; the presidency), we design the following coverage constraints:

• •

  POS Coverage (POSC): All words with POS tags as nouns (N), verbs (V), adjectives (JJ), and adverbs (RB) should be part of at least one extraction. E.g. the words Obama, gained, popularity, Oprah, endorsed, presidency must be covered in the set of extractions.

• •

  Head Verb Coverage (HVC): Each head verb should be present in the relation span of some (but not too many) extractions. E.g. (Obama; gained; popularity), (Obama; gained; presidency) is not a comprehensive set of extractions.

• •

  Head Verb Exclusivity (HVE): The relation span of one extraction can contain at most one head verb. E.g. gained popularity after Oprah endorsed is not a good relation as it contains two head verbs.

• •

  Extraction Count (EC): The total number of extractions with head verbs in the relation span must be no fewer than the number of head verbs in the sentence. In the example, there must be at least two extractions containing head verbs, as the sentence itself has two head verbs.

• •

  POSC - To ensure that the $n^{th}$ word is covered, we compute its maximum probability ($posc_{n}$) of belonging to any extraction. We introduce a penalty if this value is low. This penalty is aggregated over words with important POS tags, $J_{posc}=\sum_{n=1}^{N}x^{imp}_{n}\cdot posc_{n}$, where

  $\displaystyle posc_{n}=1-\max_{m\in[1,M]}\left(\max_{k\in\{\text{S,R,O}\}}Y_{mn}(k)\right)$

• •

  HVC - A penalty is imposed for the $n^{th}$ word, if it is not present in relation of any extraction or if it is present in relation of many extractions. This penalty is aggregated over head verbs, $J_{hvc}=\sum_{n=1}^{N}x^{hv}_{n}\cdot hvc_{n}$, where $hvc_{n}=\left|1-\sum_{m=1}^{M}Y_{mn}(R)\right|$.

• •

  HVE - A penalty is imposed if the relation span of an extraction contains more than one head verb. This penalty is summed over all extractions. I.e., $J_{hve}=\sum_{m=1}^{M}hve_{m}$, where

  $\displaystyle hve_{m}=\max\left(0,\left(\sum\limits_{n=1}^{N}x^{hv}_{n}\cdot Y_{mn}(R)\right)-1\right)$

• •

  EC - $ec_{m}$ denotes the score $\in[0,1]$ of the $m^{th}$ extraction containing a head verb, i.e. $ec_{m}=\max_{n\in[1,N]}\left(x^{hv}_{n}\cdot Y_{mn}(R)\right)$. A penalty is imposed if the sum of these scores is less than the actual number of head verbs in the sentence.

  $\displaystyle J_{ec}=\max\left(0,\sum_{n=1}^{N}x^{hv}_{n}-\sum_{m=1}^{M}ec_{m}\right)$

Ideally, no constraint violations of HVC and HVE would imply that EC would also never gets violated. However, as these are soft constraints, this scenario is never materialized in practice. We find that our model performs better and results in fewer constraint violations when trained with POSC, HVC, HVE and EC combined. The full loss function is $J=J_{CE}+\lambda_{posc}J_{posc}+\lambda_{hvc}J_{hvc}+\lambda_{hve}J_{hve}+\lambda_{ec}J_{ec}$, where $\lambda_{\star}$ are hyperparameters. We refer to the OpenIE extractor trained using this constrained loss as Constrained Iterative Grid Labeling OpenIE Extractor (CIGL-OIE).

The model is initially trained without constraints for a fixed warmup number of iterations, followed by constrained training till convergence.

Coordinated conjunctions (CC) are conjunctions such as “and”, “or” that connect, or coordinate words, phrases, or clauses (they are called the conjuncts). The goal of coordination analysis is to detect coordination structures — the coordinating conjunctions along with their constituent conjuncts. In this section we build a novel coordination analyzer and use its output downstream for OpenIE.

Sentences can have hierarchical coordinations, i.e., some coordination structures nested within the conjunct span of others Saha and Mausam (2018). Therefore, we pose coordination analysis as a hierarchical labeling problem, as illustrated in Figure 4. We formulate a 2-D grid labeling problem, where all coordination structures at the same hierarchical level are predicted in the same row.

Specifically, we define a grid of size $(M,N)$, where $M$ is the maximum depth of hierarchy and $N$ is the number of words in the sentence. The value at $(m,n)^{th}$ position in the grid represents the label assigned to the $n^{th}$ word in the $m^{th}$ hierarchical level, which can be CC (coordinated conjunction), CONJ (belonging to a conjunct span), or N (None). Using IGL architecture for this grid gives an end-to-end Coordination Analyzer that can detect multiple coordination structures, with two or more conjuncts. We refer to this Coordination Analyzer as IGL-CA.

For a conjunctive sentence, CIGL-OIE’s confidence values for extractions will be with respect to multiple simple sentences, and may not be calibrated across them. We use a separate confidence estimator, consisting of a BERT encoder and an LSTM decoder trained on (sentence, extraction) pairs. It computes a log-likelihood for every extraction w.r.t. the original sentence — this serves as a better confidence measure for OpenIE6.

We train OpenIE6 using the OpenIE4 training dataset used to train IMoJIE⁵⁵5Available from github:dair-iitd/imojie. It has 190,661 extractions from 92,774 Wikipedia sentences. We convert each extraction to a sequence of labels over the sentence. This is done by looking for an exact string match of the words in the extraction with the sentence. In case there are multiple string matches for one of the arguments of the extraction, we choose the string match closest to the other arguments. This simple heuristic covers almost 95% of the training data. We ignore the remaining extractions that have multiple string matches for more than one argument.

We implement our models using Pytorch Lightning Falcon (2019). We use pre-trained weights of “BERT-base-cased”⁶⁶6github:huggingface/transformers for OpenIE extractor and “BERT-large-cased”\@footnotemark for coordination analysis. We do not use BERT-large for OpenIE extractor as we observe almost same performance with a significant increase in computational costs. We set the maximum number of iterations, $M$=5 for OpenIE and $M$=3 for Coordination Analysis. We use the SpaCy POS tagger⁷⁷7https://spacy.io for enforcing constraints. The various hyper-parameters used are mentioned in Appendix B.

For each system, we report a final F1 score using precision and recall computed by these scoring functions. OpenIE systems typically associate a confidence value with each extraction, which can be varied to generate a precision-recall (P-R) curve. We also report the area under P-R curve (AUC) for all scoring functions except Wire57-C, as its matching algorithm is not naturally compatible with P-R curves. We discuss details of these four metrics in Appendix A.

For determining the speed of a system, we analyze the number of sentences it can process per second. We run all the systems on a common set of 3,200 sentences Stanovsky et al. (2018), using a V100 GPU and 4 cores of Intel Xeon CPU (the non-neural systems use only the CPU).

We examine extractions from a random sample of 50 sentences from CaRB validation set, as output by OpenIE6. We identify three major sources of errors in these sentences:
Grammatical errors: (24%) We find that the sentence formed by serializing the extraction is not grammatically correct. We believe that combining our extractor with a pre-trained language model might help reduce such errors.
Noun-based relations: (16%) These involve introducing additional words in the relation span. Although our model can introduce [is], [of], [from] in relations (Section 3), it may miss some words for which it was not trained. E.g. [in] in (First Security; based [in]; Salt Lake City) for the phrase Salt Lake City-based First Security.
Lack of Context: (10%) Neural models for OpenIE including ours, do not output extraction context Mausam et al. (2012). E.g. for “She believes aliens will destroy the Earth”, the extraction (Context(She believes); aliens; will destroy; the Earth) can be misinterpreted without the context.

We also observe incorrect boundary identification for relation argument (13%), cases in which coordination structure in conjunctive sentences are incorrectly split (11%), lack of coverage (4%) and other miscellaneous errors (18%).

We propose a new OpenIE system – OpenIE6, based on the novel Iterative Grid Labeling architecture, which models sequence labeling tasks with overlapping spans as a 2-D grid labeling problem. OpenIE6 is 10x faster, handles conjunctive sentences and establishes a new state of art for OpenIE. We highlight the role of constraints in training for OpenIE. Using the same architecture, we achieve a new state of the art for coordination parsing, with a 12.3 pts improvement in F1 over previous analyzers. We plan to explore the utility of this architecture in other NLP problems. OpenIE6 is available at https://github.com/dair-iitd/openie6 for further research.

We thank the anonymous reviewers for their suggestions and feedback. Mausam is supported by IBM AI Horizons Network grant, an IBM SUR award, grants by Google, Bloomberg and 1MG, Jai Gupta Chair Fellowship and Visvesvaraya faculty award by Govt. of India. We thank IIT Delhi HPC facility for compute resources. Soumen was partly supported by a Jagadish Bose Fellowship and an AI Horizons Network grant from IBM.