Learning Semantic Role Labeling from Compatible Label Sequences

Semantic Role Labeling (SRL, Palmer et al., 2010) aims to understand the role of words or phrases in a sentence. It has facilitated other natural language processing tasks including question answering FitzGerald et al. (2018), sentiment analysis Marasović and Frank (2018), information extraction Solawetz and Larson (2021), and machine translation Rapp (2022).

Semantic role labeling can take various forms, each associated with different datasets. Predicates can be coarsely divided into PropBank Palmer et al. (2005) senses, each with a core set of numbered semantic arguments (e.g., Arg0–Arg5). There are also modifier arguments (e.g., ArgMLoc) typically representing information such as the location, purpose, manner or time of an event. Alternatively, predicates can also be hierarchically clustered into VerbNet Schuler (2005) classes according to similarities in their syntactic behavior. Each class admits a set of thematic roles (e.g. Agent, Theme) whose interpretations are consistent with all predicates within the class.

As a modeling problem, SRL requires associating argument types and phrases with respect to an identified predicate. The two labeling tasks (i.e., VerbNet SRL and PropBank SRL) are closely related; but they differ in their treatment of predicates and have disjoint label sets. Learning jointly can improve data efficiency across the different labeling SRL tasks.

A common formulation used to instantiate this idea in prior work is multitask learning (e.g. Strzyz et al., 2019; Gung and Palmer, 2021): each label set is treated as a separate labeling task, and sometimes also modeled with inter-task feature interaction or consistency losses. While multitask learning often works well in such cases, the loss formulation represents a conservative view over label compatibilities of different tasks. At prediction time, subtask modules still run independently and are not constrained by each other. Consequently, decoded labels may violate structural constraints with respect to each other. In such settings, constrained inference (e.g., Fürstenau and Lapata, 2012; Greenberg et al., 2018) has been found helpful. However, this raises the question of how to involve such inference during learning for better data efficiency. Furthermore, given the wider availability of PropBank-only data (e.g., Pradhan et al., 2013, 2022), how to efficiently benefit from such data also remains a question.

In this paper, we argue that the two disjoint but compatible labeling tasks can be more effectively modeled as one task using their compatibility structures that are already explicitly defined in the form of Semlink Stowe et al. (2021). Semlink offers mappings between various semantic ontologies including PropBank and VerbNet. Gung and Palmer (2021) devised a deterministic conversion from PropBank label sequences to VerbNet ones using only the unambiguous mappings in Semlink. This conversion gives a test bed that has half of the predicates in CoNLL05 SRL dataset Carreras and Màrquez (2005) with both VerbNet and PropBank jointly labeled.

Given this setting, we propose a simple and effective joint CRF model for the VerbNet SRL and PropBank SRL tasks. In addition to the joint CRF, we propose an inference constraint that uses compatible label structures defined in Semlink, and show that our constrained inference achieves higher overall SRL F1—the average of VerbNet and ProbBank F1 scores—than the current state-of-the-art. Indeed, when PropBank labels are observed, it achieves over $99$ F1 on the VerbNet SRL, suggesting the possibility of an automated annotation helper. We show that our formulation naturally extends to a constrained marginal model that learns from the more abundant PropBank-only data in a semi-supervised setting. When learning and predicting with constraints, it achieves even better SRL F1 in out-of-domain generalization.¹¹1The code for reproducing our experiments: https://github.com/utahnlp/marginal_srl_with_semlink

We consider modeling VerbNet (VN) and PropBank (PB) SRL as a joint labeling task. Given a sentence $x$, we want to identify a set of predicates (e.g., verbs), and for each predicate, generate two sequences of labels, one for VerbNet arguments $y^{V}$, the other for PropBank arguments $y^{P}$. With respect to VN parsing, a predicate is associated with a VerbNet class that represents a group of verbs with shared semantic and syntactic behavior, thereby scoping a set of thematic roles for the class. Similarly, the predicate is associated with a PropBank sense tag that defines a set of PB core arguments along with their modifiers. A example is shown in Tab. 1.

We treat predicate classification and argument labeling as separate tasks and focus on the latter.²²2Prior work (e.g., Täckström et al., 2015) has shown that the predicate disambiguation can be modeled with high accuracy using standalone classifiers. Assuming predicates $u$ and their associated VN classes $\eta$ and PB senses $\sigma$ are given along with $x$, we can write the prediction problem as:

$\displaystyle(x,u,\eta,\sigma)\rightarrow(y^{V},y^{P})$

(1)

To eliminate the errors discussed in Sec. 2.2, we propose to model the disjoint SRL tasks using a joint set of labels. This involves converting multitask modeling into a single sequence labeling task whose labels are pairs of PB and VN labels. Doing so not only eliminates the BIO inconsistency, but also exposes an interface for injecting Semlink constraints.

Our model uses RoBERTa Liu et al. (2019) as the backbone to handle textual encoding, similarly to the SRL model of Li et al. (2020). At a high level, we use a stack of linear layers with GELU activations Hendrycks and Gimpel (2016) to encode tokens to be classified for a predicate. For the problem of predicting arguments of a predicate $u$, we have an encoding vector $\phi_{u,i}$ for the $i$-th word in the input text $x$.

	$\displaystyle e$	$\displaystyle=\texttt{map}(\textsc{RoBERTa}(x))$		(3)
	$\displaystyle\phi_{u}$	$\displaystyle=\left\{f_{ua}\left([f_{u}(e_{u}),f_{a}(e_{i})]\right),\forall_{i}\in x\right\}$		(4)

Here, map sums up word-piece embeddings to form a sequence of word-level embeddings, the functions $f_{u}$ and $f_{a}$ are both linear layers, and $f_{ua}$ denotes a two-layer network with GELU activations in the hidden layer. We use a dedicated module of the form in Eq. 4 for the VN and PB subtasks. This gives us a sequence of vectors $\phi_{u}^{v}$ for VN and a sequence of vectors $\phi_{u}^{p}$ for PB.

Next, we project the VN and PB feature sequences into a $|Y^{V}|\times|Y^{P}|$ label space:

$\displaystyle z_{u}$

$\displaystyle=\{g([\phi_{u,i}^{V},\phi_{u,i}^{P}]),\forall_{i}\in x\}$

(5)

Here, $g$ is another two-layer GELU network followed by a linear projection that outputs $|Y^{V}|\times|Y^{P}|$ scores, corresponding to VN-PB label pairs. The final result $z_{u}$ denotes a sequence of VN-PB label scores for a specific predicate $u$. In addition, we use a CRF as a standard first-order sequence model over $z_{u}$ (treating it as emission scores), and use Viterbi decoding for inference. The training objective is to maximize:

$\displaystyle\log P(y^{VP}|x)$

$\displaystyle=s(y^{VP},x)-\log Z(x)$

(6)

where $s(\cdot)$ denotes the scoring function for a label sequence that adds up the emission and the transition scores, and the term $Z(x)$ denotes the partition that sums exponentiated scores over all label sequences. The term $y^{VP}$ denotes the label sequence that has VN and PB jointly labeled. We will refer to this model as the joint CRF, and the label sequence as the joint labels.

Compared to data with both VerbNet and PropBank fully annotated, there is more data with only PropBank labeled. The Semlink corpus helps in unambiguously mapping $\sim 56\%$ of the CoNLL05 data Gung and Palmer (2021). Therefore, a natural question is: can we use PropBank-only data to improve the joint task?

Here, we explore model variants based on the joint CRF architecture described in Sec. 3. We will focus on modeling for PropBank-only sentences.

Here we discuss the implementation of the $y_{u}^{\textsc{seml}}$. Remember that each pair of VerbNet class $\eta$ and PropBank sense $\sigma$ uniquely determines a set of joint argument labels for the predicate $u$. For brevity, let us denote this set as $\textsc{seml}(u)$ (e.g., Tab. 1). Eventually, we want the Viterbi-decoded label sequence to comply with $\textsc{seml}(u)$. That is,

	$\displaystyle\forall(l^{V},l^{P})\in$	$\displaystyle\textsc{seml}(u)\rightarrow$
	$\displaystyle\forall i,$	$\displaystyle\Big{[}\big{(}\forall_{l\notin\textsc{seml}(u)}(y_{i}^{V}=l^{V},l)\notin y_{u}^{VP}\big{)}$
		$\displaystyle\wedge\big{(}\forall_{l\notin\textsc{seml}(u)}(l,y_{i}^{P}=l^{P})\notin y_{u}^{VP}\big{)}\Big{]}$		(9)

where $i$ denotes the location in a sentence, $y_{u}^{VP}$ is the joint label sequence, consisting of $(y_{i}^{V},y_{i}^{P})$ pairs. The constraint in Eq. 9 translates as: if a VerbNet argument is present in the predicate $u$’s Semlink entry, we prevent it from aligning to any PropBank arguments not defined in Semlink; and the same applies to PropBank arguments.

This constraint can be easily implemented by a masking operation on the emission scores of the joint CRF, thus can be used at both training and inference time. During inference, it effectively ignores those label sequences with Semlink violations during Viterbi decoding:

$\displaystyle y^{VP}_{u}$

$\displaystyle=\arg\max_{y\in y_{u}^{\textsc{seml}}}s(y)$

(10)

In Sec. 6, we will show that using Eq. 10 always improves the overall SRL F1 scores.

In this section, we aim to verify whether the compatibility structure between VerbNet and PropBank (in the form of Semlink) has a positive impact on their sequence labeling performances.

We report statistical metrics in Sec. 7.1. In Sec. 7.2, we analyze the use of constrained learning on the jointly labeled data.

The use of constraints in SRL has a long history that mostly focuses on the PropBank SRL task. Earlier work investigated inference with constraints (e.g. Punyakanok et al., 2004; Surdeanu et al., 2007; Punyakanok et al., 2008). Other work developed constrained models for learning (e.g. Chang et al., 2012) ; or incorporated constraints with emerging neural models (e.g. Riedel and Meza-Ruiz, 2008; Fürstenau and Lapata, 2012; Täckström et al., 2015; FitzGerald et al., 2015; Li et al., 2020).

VerbNet SRL, on the other hand, is often studied as a comparison or a helper for PropBank SRL Kuznetsov and Gurevych (2020). Yi et al. (2007) showed that the mapping between VerbNet and PropBank can be used to disambiguate PropBank labels. It has been shown that model performance on VerbNet SRL is affected more by predicate features than PropBank SRL Zapirain et al. (2008). In a sense, our observation that VerbNet F1 gains larger improvements from Semlink is also consistent with prior work. Beyond comparison work, Kazeminejad et al. (2021) explored the downstream impact of VerbNet SRL and showed promising uses in entity state tracking.

We thank the reviewers for their helpful insights and comments. We gratefully acknowledge the support of NSF under grants #1801446 (SATC) and #1822877 (Cyberlearning), and of DARPA CwC (subcontracts from UIUC and SIFT), DARPA AIDA Award FA8750-18-2-0016 (RAMFIS), DARPA KAIROS FA8750-19-2-1004-A20-0047-S005 (RESIN, sub to RPI, PI: Heng Ji), as well as DTRA HDTRA1-16-1-0002/Project 1553695 (eTASC - Empirical Evidence for a Theoretical Approach to Semantic Components).

The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies of any government agency.