Thematic Fit Bits: Annotation Quality and Quantity Interplay
for Event Participant Representation

Our first goal is to revisit a widely held assumption in the NLP community: mediocre machine-predicted annotation yields at scale better (or equivalent) models than high quality annotations (manual or state-of-the-art machine-predicted) whose scale is much smaller, due to compute, cost, and time constraints. ?) and ?), inter alia, use self-training¹¹1Augmenting few manual annotations with ones predicted by a prior version of the same parser over a large text. to improve syntactic parsing – as an alternative to manually annotating more data – in same or different domain/genre. ?) show that using 100k machine-predicted constituency parses to train a new dependency parser contributed the equivalent of 2k manually annotated parses. Manually annotating is slower and more expensive but better by definition.

However, despite growing amounts of annotated and unannotated textual resources, a number of tasks with traditional linguistic levels of representation remain a challenge, with unsatisfactory performance in various research areas and applications: artificial intelligence (AI), machine reading / knowledge graph population, chatbots and natural language understanding (NLU), as well as computational psycholinguistic modeling and computational linguistics.

How do quality and quantity affect our above-mentioned semantic and psycholinguistic tasks?

Our second goal is to explore ways to improve semantic modeling and the representations used for this modeling. We use semantic modeling to refer to tasks at the intersection of NLP and psycholinguistics that have to do with representing and processing generalized event knowledge [Pustejovsky, 1991, Zarcone and Padó, 2011] . The underlying tasks include:

Semantic role labeling (SRL) is the task of annotating text according to semantic frames and their roles as defined in frameworks such as FrameNet, VerbNet, or PropBank [Baker et al., 1998, Schuler and Palmer, 2005, Palmer et al., 2005]. For example, given ‘I cut the cake with…’ (1) ‘marzipan’ or (2) ‘a knife’, the role Instrument/A3-MNR is normally desired for (2) but not for (1).

Role prediction: a simplified SRL task we use here. Instead of a sentence, the input is just a verb $v$ and a noun $n$; optionally additional nouns and their thematic relation (role) with $v$; the expected output is the most likely thematic role of $n$ with $v$ in the same event.²²2A similar task – only without the input nouns – is the (role) selectional preference of the verb: what roles are more likely with this verb? E.g., ‘cut’: Agent/Arg0, Patient/Arg1, Instrument/Arg3-MNR.

Slot / Role filling:³³3We interchangeably refer to it here as word prediction. given a predicate (typically a verb) and a thematic role, what word or phrase would be most appropriate for that role? This task can be viewed as the complement of SRL (given the word or phrase, what is its role?).

Thematic fit: Given a predicate and a role (say, ‘cut’ + Instrument), how well would a speaker of a given language (here, English) find ‘knife’ or ‘spoon’ fitting to the given role? And by extension: given a subset of a predicate and arguments (optionally also modifiers), can we predict the typicality level of the most recently added member to the rest of the given subset? This is often an abstraction of sentence comprehension (in humans): our thematic fit estimation changes as we hear more of the uttered sentence [Amsel et al., 2015].

Indirect thematic fit estimation learning from SRL annotations has shown promising results [Hong et al., 2018, Tilk et al., 2016, Santus et al., 2017]. Following ?) and others, we consider only the arguments’ syntactic heads together with their semantic roles.⁴⁴4We also follow them in using the simplified ProbBank roles (A0, A1, …), unlike much of the thematic fit literature that uses Agent, Patient, etc. [Dowty, 1991]. Given a new role, predict the fitness level of all known words and use the score of the given filler in the full score distribution as a fitness rating. (We also look at the complement: given a new word, predict the fitness of each possible role). These predictions are scored relative to human judgments (see Section 2).

Exploring the data requirements of modeling human semantic representations allows us to revisit the question of the inherent difficulty of semantic tasks: does semantic processing simply require more annotated data to achieve high quality, or has it not been represented in a way conducive for computers to learn adequately?

We look into how annotation quality and quantity (both the number of semantic frames and the number of sentences) affect learning. We also explore annotation granularity, taking thematic role set granularity (number of roles in model) as a test case. We often see that modelers focus on only the two most frequent PropBank semantic arguments (Arg0 and Arg1) and ignore the rest or lump the rest together under a catch-all tag. Similarly, they focus on only few modifiers (e.g., ?) and ?) use 2 core roles and 3 modifiers). We therefore trained models with increasing numbers of thematic role types (the predicate, its core arguments, and often-optional, non-core arguments or modifiers), using the better taggers, parsers, and labelers, and observed changes in prediction quality.

To summarize our main contributions, we

1. 1.

   test how quality of annotation affects supervised role/word prediction, as training set size increases. We show in small to large sizes that the mediocre annotation method is not as useful as better quality annotation.

2. 2.

   test how quality of annotation affects thematic fit estimation as an application of our models that is not part of the training objective. We show that quality increases with training size but surprisingly, variance is high even in larger sizes, leaving no clear winner annotation.

3. 3.

   claim that the high variance of the (indirectly optimized for) thematic fit estimation makes it more difficult to interpret conclusions from previous studies that did not report it.

4. 4.

   show new state-of-the-art results on role and word prediction, as well as thematic fit estimate correlations.

5. 5.

   tease apart effects of quality and quantity; tease apart the number of training sentences from the number of training frames (the semantic frames annotated in these sentences).

6. 6.

   test how annotation granularity (role set size) affects thematic fit (and role/word prediction)

7. 7.

   provide a new, open-data, large lexical semantic (and syntactic) resource in English, revising and expanding the previously published RW-Eng [Sayeed et al., 2018].

The v1 corpus [Sayeed et al., 2018] consists of the SENNA-derived SRL output over 78M sentences from 2.3M documents. The documents come from the BNC and ukWaC. SENNA extracts multiple predicates per sentence and, for each predicate, it identifies spans of text representing noun phrases that fill PropBank roles for that predicate. For every document, sentence, and predicate in that sentence, v1 contains XML-formatted information on the corresponding SENNA output. In particular, it uses a series of head-finding heuristics [Sayeed et al., 2016] to identify the syntactic heads of the role-filling spans—typically noun phrases, which can contain complex constituents such as subordinate clauses, but the SRL role spans could also cover only fractions of syntactic constituents, hence the need for a heuristic beyond only using a parser.

NLP often contains “pipelines” of serial annotation processes, such as: tokenization and morphological analysis (including lemmatization or stemming), syntactic parsing, and a final processing such as machine translation, NLU (for chatbots), and sometimes SRL. As mentioned in Section 1, until about ten years ago, a rule of thumb often held: better quality of intermediate processing results in better quality at the end, although improvements are not linear, and small intermediate gains do not always translate to gains at the end of the pipeline.

Here we set to test out the new rule-of-thumb of the last decade for the case of thematic role prediction, slot filling (word prediction), and thematic fit estimation. We replaced annotation layers from older tools with annotations based on more recent tools (see below), introducing a non-negligible improvement in intermediate annotation quality.

The first step in doing so is to determine a consistent tokenization schema across all annotation layers, as some taggers either expect a certain schema or apply their own even if input text is largely tokenized. We iteratively modified our tokenization schema to reduce token count mismatch between the new layers from over 20% of sentences to less than 1%. Once we reached this low mismatch ratio, we marked and excluded the fewer mismatched cases from our experiments.

We added the following new annotation layers:

Of about 3500 files, few (less than 0.4%) were discarded due to processing issues, leaving us with 3490 files. Due to the fact that the corpus is comprised of more than one source, we assigned 16 files as the development set, and 16 as the test set, chosen uniformly⁷⁷7Dev files: [ 217, 435, 651, 868, 1085, 1302, 1519, 1736, 1953, 2170, 2387, 2604, 2821, 3038, 3255, 3472]. Test files: [ 218, 436, 652, 869, 1086, 1303, 1520, 1737, 1954, 2171, 2388, 2605, 2822, 3039, 3256, 3473]. . The rest of the files were used as the full training set. Subsets of this training set were each chosen uniformly too, to emulate availability of smaller training sets. This split departs from ?), which used the last 0.4% (14 files) as the test set, the immediately preceding 0.4% as the dev set, and the rest for training.

We also tested our models on the above-mentioned thematic fit test sets, without optimizing on them:

Padó-all: A human-rated thematic fit score dataset collected with psycholinguistic motivations, created by ?) and containing 414 verb-noun-role triplets, where every two triplets differ only in the role, one of {Arg0, Arg1, Arg2}.

McRae-all: A similar dataset with human scores, created by ?), containing 1,444 such triplets, grouped in pairs similarly, but the roles are only {Arg0, Arg1}, and the words are less frequent (a harder task).

For our baseline (v1-based models), we used a multi-task residual network (ResNet) model [He et al., 2016, Jégou et al., 2010]. Our implementation is similar to the best reported model in ?), called ResRofa-MT, for ease of comparison.⁸⁸8We thank Xudong Hong for his help. One task was role prediction, given a verb, a (typically noun) word, and zero or more $\langle$role,word$\rangle$ pairs. For example, given ‘cut’ and ‘knife’, predict Instrument, or in PropBank’s roles: A3-MNR. A second task was word prediction (slot / role filling), given the word’s role, the verb, and the zero or more $\langle$role,word$\rangle$ pairs. For example, given ‘cut’ and A3-MNR, predict ‘knife’. For both tasks we could optionally provide more input, say, $\langle$A1,‘cake’$\rangle$. Aside from having different prediction layer per task, the tasks shared the same neural network (and parameters). Apart from software engineering differences, the most notable difference in our implementation is having two separate labels for missing role and unknown role instead of one for both. The former is used to mark the absence of a certain role from the annotated frame instance. The latter is used as a catch-all for sparser roles not explicitly represented. The baseline role set was comprised of PRD, Arg0, Arg1, ArgM-TMP, ArgM-LOC, ArgM-MNR: the predicate, PropBank’s arguments 0 and 1, the temporal, location, and manner modifiers (and missing role and unknown role).

For faster training time, we used a batch size of 1024 samples (unless otherwise specified) with the risk of too coarse updates. We kept a simple setting of 0.1 learning rate and no decay. We applied the same vocabulary pruning to the top 50k most frequent lemma forms as ?) did.

We trained and evaluated models in the above configuration with increasing training set size (see Section 3.3), measured in percentage of total number of available training sentences. For each training set size, we trained few models on v1  and same number of models on v2.

Table 1 shows that role prediction accuracy increases with training set size as expected and surpasses 90% at 1% of the training set for v1 and already at 0.1% for v2. At 10% of the training, it reaches mid-90s for v1 and high-90s for v2, with small further gains at 20%.⁹⁹9The full dataset size was prohibitive for training given our computing resources. The advantage of v2 was kept throughout but decreased from over 5-6% (absolute) at 0.1% size to 3-4% at higher training set sizes. Word prediction accuracy followed similar trends, but v2–v1 gaps there remained about 6% in all set sizes.¹⁰¹⁰10Preliminary experiments, having varying sized of dev and test sets, showed the same pattern: v2 advantage.

We also tested the models on two thematic fit tasks on which the models were not trained, using the additional test sets described in Section 3.4. Variance over multiple training runs per model was not reported in the relevant literature, so the high variance we see on both Padó-all¹¹¹¹11In Padó-all evaluations, test set target Arg2 was mapped to unknown-role, since it was not in the model’s role set (McRae-all only has Arg0,Arg1 targets). and McRae-all is a novel finding.¹²¹²12We trained 5 models for each version in small training sizes, and 3 for each in large sizes, due to resource limitations. We report both the best- and last-epoch results for these tasks on the rightmost two columns of Table 1. We see v2 advantage on Padó-max 10% and 20% but not at lower sizes. Padó-final results are mixed, particularly at 10%-training, as are the results on McRae (both last and best). We see clear training size effects on McRae(best+last), but not on Padó (except in the smallest size).

The v1  results on the 10% and 20% subsets are our closest replication of ?), modulo the above-mentioned role representation.

Model design includes decisions about features and their granularity. Generally, fine-grained features provide sharper, more accurate distributions of underlying phenomena, but their values, especially at the distribution “tail”, suffer from higher sparsity, which may lead to difficulty in learning them well and hence lower performance overall. This tension between granularity and sparsity exists also for the case of role set granularity. Many studies have chosen the coarser side, using only few thematic roles. For example, ?), which we take as our main baseline, use only 2 arguments and 3 modifiers, within the PropBank framework—in itself already a framework with a coarse role set (compared to VerbNet and FrameNet).

We tested the effect of increasing role set granularity. In other words: is less (less coarse role set) more (higher quality)? We analyzed the distribution of the roles in the dev set (see Table 3) and expanded the role set one role at a time, according to their frequency (more frequent first). For training run time economy, all models in this subsection were trained on 1% of the v2 data.

The first semantic role to be added was Arg2. Note that now in Padó-all evaluations, Arg2 is no longer mapped to unknown-role. See Table 2. It turns out that while role set prediction accuracy slightly dropped, word prediction accuracy actually improved by more than 1%. Same trend held also for Padó-all (about 3% gain in Spearman’s correlation) and McRae-all (over 1%). This stands in contrast to preliminary experiments on v1.¹³¹³13Hong, personal communication.

Next down the role list, we added ArgM-MOD. While we saw a slight gain in word prediction accuracy, we saw a drop in Padó-all and perhaps a slight drop in McRae-all. However, adding ArgM-ADV resulted in a drop in role prediction but gains in the thematic fit tasks. Adding ArgM-DIS resulted in some drop on the thematic fit tasks, while adding ArgM-NEG yielded relative gains in word prediction and Padó-all. Adding all roles to the model (all.args+mods) yielded mixed results: a further small drop in role prediction (in fact, no model outperformed the baseline on this task), a pronounced gain in word prediction (highest result even compared to Table 1!), and lower scores on the thematic fit tasks compared to adding only Arg2.

Modifiers are often more freely optional, and are not considered core arguments of the semantic frame. We therefore also tested the effects of only adding core arguments to the baseline roleset (see bottom part of Table 2). Adding Arg3 resulted in no much change in any task, compared to +Arg2. Adding Arg4 yielded a gain on Padó-all last result (but not the maximal result). Adding Arg5 resulted in a clear drop on the thematic fit tasks, which is expected: Arg5 is very sparse, so its learnability is low, and roleset confusability is higher due to increased number of roles. However, we take differences on Padó-all and McRae-all with a grain of salt, given the above-mentioned high variability.