Cis²: A Simplified Commonsense Inference Evaluation for Story Prose

Hwang et al. (2021) formalized the commonsense inference task (CI) for AI systems as a knowledge three-tuple, to predict the object of a relation given the subject and relation. This formulation of commonsense inference can be structured as a graph, where the subjects and objects are nodes and the relations are the edges connecting the entities. These commonsense knowledge graphs (CKGs) explicitly encode the structure of inference relationships between entities. ATOMIC Sap et al. (2019) is one such CKG dataset that organizes everyday events into if-then relationships. COMET Bosselut et al. (2019) is a transformer language model designed on top of ATOMIC relations, showing language models can encode and generalize commonsense information.

However, Wang et al. (2021) show that language models struggle to perform generalizable commonsense inference across three popular CKG datasets: ConceptNet Speer et al. (2017), TupleKB Dalvi Mishra et al. (2017), and ATOMIC Sap et al. (2019). They found that LMs trained on several CKGs have limited ability to transfer knowledge to unseen CKGs, and that adaptation generalizes well to unseen subjects, but less so on unseen objects.

Although these graphs do well at representing facts and their relations, their statements lack context and would need to be adapted to a textual domain, such as story prose. Using them to generate a story as-is would fail to engage readers since the “story” would simply be a series of facts. Our work goes beyond the explicit structure of CKGs, focusing on finding and leveraging commonsense relations in natural language short stories.

Early research on automated story generation research focused on designing systems that create coherent stories Lebowitz (1985); Turner and Dyer (1985); Liu and Singh (2002); Young et al. (2013). Despite the success of neural networks for AI tasks, commonsense and coherence remain big issues for story generation systems.

Applying commonsense reasoning to the events of a story has been proposed as one way to tackle the difficult problem of assessing the quality of machine-generated stories. The Story Cloze Test Mostafazadeh et al. (2016) formulates story ending generation as a multiple-choice task, having systems look at several possible endings and predict the one that is most reasonable. Guan et al. (2019) integrated commonsense reasoning directly into their Story Cloze model by building context clues and using implicit knowledge.

Commonsense reasoning can also help story generation with issues in plot coherence. Martin (2021) created a neurosymbolic system that leveraged VerbNet Brown et al. (2019) facts to ground neural story generation in commonsense reasoning. They did this by tracking the story state and pruning out impossible options that a neural network provided as candidate next sentences for the story. Similarly, the Commonsense inference Augmented neural StoryTelling (CAST) framework Peng et al. (2021) modeled interactions between multiple characters using ATOMIC. The stricter, more explicit generation constraints of CAST produced more coherent and on-topic two-character stories than generating via sampling from a distribution alone.

TellMeWhy Lal et al. (2021) is a dataset built on top of ROCStories Mostafazadeh et al. (2016), consisting of 30k questions on why characters perform their actions and the corresponding answers. They found that current state-of-the-art models performed far worse than humans, especially on questions whose answers are external to the narratives. This contrasts with the findings discussed in Mostafazadeh et al. (2020) that language models can approach human performance.

GLUCOSE addresses the task of predicting relationships between statements explicitly or implicitly expressed within a text, a task we term contextual commonsense inference (CCI). An example GLUCOSE entry can be found in Table 2. The entries are organized to reflect the CCI task and are formalized as input-output tuple pairs, with input tuple

$$\displaystyle\langle\text{{\color[rgb]{0,0,1}story {S}, selected sentence {X}, dimension {D}}}\rangle,$$

(2)

where a story S consists of five sentences [s₀, s₁, s₂, s₃, s₄], the selected sentence X is the sentence on which the rule is centered, and the number dimension D is one of the ten dimensions from Table 1—and output tuple

$$\displaystyle\langle\text{{\color[rgb]{0.5,0.5,0}specific rule {R\textsubscript{S}}, general rule {R\textsubscript{G}}}}\rangle,$$

(3)

where the specific rule R_S is the relation between X and Y. Y can be either (1) another sentence in the story or (2) an implicit statement from outside the text. The general rule R_G is the same rule as R_S but using generalized tags for named entities (e.g., Someone_A instead of Fred). To summarize, the GLUCOSE task is: given S, X, and D, predict/generate R_S and R_G.

In this paper, we compare to their best model, a finetuned T5 model Raffel et al. (2020), which achieved a 71.26 average SacreBLEU Post (2018) across the 10 dimensions on predicting general rules and a 75.65 average for the specific rules.¹¹1Our best-effort replication of their experiments achieves slightly lower BLEU scores (66.2 & 70.7, respectively) due to resource limitations (detailed in Appendix A.4). The models were also rated for “correctness” using crowdsourcing, where their T5 model scored 2.5/3 averaged across all 10 dimensions on a 4-point Likert scale mapped to a numerical scale of 0-3. For context, their closest baseline got a 2.21/3 average and the gold standard was 2.8/3.

We find that the GLUCOSE dataset is well-designed and of good annotation quality. However, we take issue with the GLUCOSE task, which asks a model to perform two tasks simultaneously: commonsense inference and language generation. Due to this conflation of tasks, the model, in generating its output, would rely heavily on the already-good language generation ability of T5 and neglect learning enough CCI. T5 Raffel et al. (2020) and other transformer LMs were designed to perform language generation tasks. Therefore, by including text generation as part of CCI, T5 will focus on paraphrasing or even copying story sentences.

There are several one-to-one correspondences between parts of the input and output in the original GLUCOSE task (illustrated in Figure 1). For example, for all GLUCOSE entries, the output contains at least one paraphrased sentence from the input. Conflation with paraphrasing worsens with BLEU as the evaluation metric, where incorrect commonsense inferences can score partial credit if they have words in common.

Table 4 compares the results of T5 models trained on the diagnostic tasks. We report test set results on the averaged dimensions 1-10, as well as averaged dimensions 1-5 (X is the second statement), and 6-10 (X is the first). Following Mostafazadeh et al. (2020), we use SacreBLEU Post (2018) with equal weights up to 4-grams. We report results for both specific and general rules, but focus on specific.

Original, of course, performs the best as its input has the most available information. History and Mask X perform similarly to each other and far worse than the other diagnostic tasks. History, with only the pre-context, has a a 35-point BLEU gap for specific rules (16 for general) compared to Original averaged across all dimensions.

Adding to History multiple sentences of the post-context gives Mask X, and modest score gains (35.9 vs 41.6 specific). However, adding to History just the one selected sentence X gives History+X, which performs very closely to Original for both specific and general rules (70.7 vs 68.3 specific). Furthermore, comparing trends between dimensions 1-5 and 6-10, we find that 6-10 scores are mostly higher, for both general and specific, than 1-5.

These results and their trends show that BLEU scores are highly contingent on having X as input over all other sentences. Conflation always occurs for X, since this is copied from the input, and conflation is also worse in cases where an incorrect statement Y was generated but contains tokens that match the correct statement. We believe it is unlikely that achieving ~35.9 BLEU on specific rules for History would mean that it is half as good at CCI than Original, with 70.7 BLEU specific. We found that the fine-tuned T5 models perform some CCI, but BLEU scores are hard to interpret and can be unreliable.

To evaluate the Cis² formulation, we need to convert story sentences into Cis² output labels, as in Equation 4. See Figure 2 for the conversion process. Each sentence of an input story corresponds to a tag <s₀> to <s₄> with indexes corresponding its position in the story. To get the three Cis² output labels, we do the following: (1) Identify selected sentence X from the input since it always be denoted as the sentence with the asterisks surrounding it. The input dimension informs the position of sentence X in the output—whether is <s_a> or <s_b>; (2) Get the relation REL from the output directly; and (3) Calculate the similarity of “other” sentence Y from the output to every other sentence in the input story and select the closest match.

To find the remaining token, we look at the specific rule from the original GLUCOSE task output, which consists of two statements separated by relation REL. We will call them P₀ and P₁. Suppose X corresponds to P₀, and we need to find which sentence Y corresponds to P₁. We do this by iterating over the sentences (excluding X), for each calculating its similarity with P₁. We take the index of the sentence with the highest similarity to P₁ as <s_b>. We describe our experiments with several sentence similarity metrics in Section 5.2.

Being a heuristic approach, generated Cis² labels are not perfect. However, our manual inspection finds most labels are reasonable for GLUCOSE entries that have an explicit Y (from the story). Cis² labels do not exist for those GLUCOSE entries with implicit relationships⁴⁴4Mostafazadeh et al. (2020) estimate these are a minority., i.e. Y is not in the original story. We attempted to filter these out by removing any training examples that did not pass a threshold⁵⁵50.16 is the mean SBERT value across the train set. of SBERT $\leq 0.16$ for any sentence in the story. However, this resulted in a slight drop in the final evaluation, so these examples were kept.

We run the conversion method on the GLUCOSE train set and train a T5 model using the same hyperparameters used for our other models with the task of generating the three-token Cis² label, given the GLUCOSE input. We refer to this model as Cis²-T5. Note that although using Cis² tags turns this into a classification problem, the model is still doing generation to predict the output.

In Section 4, we showed that BLEU is not an appropriate metric for the CCI task, given the GLUCOSE models’ extensive copying and paraphrasing. Furthermore, Cis²-T5 generates Cis² tags instead of full sentences, making it non-trivial to compare to the Original GLUCOSE T5 model.

We run the conversion method from Section 5.1 on each model’s specific rule output to obtain its predicted Cis² labels, and on the GLUCOSE test set to obtain the Cis² test set.⁶⁶6For future work we plan to obtain ground-truth test labels via crowdsourcing. Both are now formatted as in Equation 4. This enables us to do an exact-match comparison between the model labels and the test set labels, and removes the associated issues with evaluating generated text. In effect, the Cis²evaluation considers requires the correct sentence Y to be chosen; there is no partial credit for those outputs that can easily be inferred from input: the selected sentence X, and REL.

The sentence similarity metric used is crucial in the process of heuristically generating Cis² labels. We experimented with both BLEU scores of lemmatized tokens, as well as Sentence-BERT (SBERT) Reimers and Gurevych (2019). By using BLEU for sentence similarity, GLUCOSE Original achieves 66.0%, whereas Cis²-T5—despite being trained on these Cis² labels converted with BLEU—only achieves 57.2% accuracy. This stems from same issues of BLEU measuring language generation, rather than CCI, as discussed in Section 4. Also, this shows that the Cis² classification task does not favor our Cis² system by default.

Therefore, for the final evaluation we opt for SBERT, a more context-dependent similarity metric. Results for this evaluation are shown in Figure 3. We compare all of our results to a random baseline which is the probability one of the 4 other story sentences is randomly selected for the index of Y; this would have an accuracy of 25% (the dashed horizontal line in Figure 3). Out of all the models, Cis²-T5 achieves the highest score at 66.2%, while Original is not far behind at 61.9%. As for the diagnostic tasks, we see the same score ordering of models with BLEU evaluation. History+X scores 8% lower than Original. History and Mask X perform even worse than random, indicating that their BLEU performance was largely due to partial token matches.⁷⁷7Experiments comparing Cis² to models that are trained to generate only specific rules can be found in Appendix A.6.

The best GLUCOSE model Original achieves 70.7 specific BLEU, but only 61.9% Cis² accuracy. Although we cannot directly compare BLEU of generated output, and Cis² exact match accuracy, we have shown that Cis² provides a fairer estimate of CCI performance of these fine-tuned T5 models by removing language generation from evaluation. These Cis² results are promising, but there is still much room for improvement.

We plan to collect ground-truth Cis² labels via crowdsourcing for the entire test set, and for some training examples. To simplify the task, we will have workers verify, and correct if necessary, the heuristic Cis² labels.

Future work can further explore utilizing GLUCOSE and related datasets for story generation tasks. One promising avenue to extending our CCI evaluation to story generation settings is incorporating our approach with the COINS framework Paul and Frank (2021), which generates contextualized inference rules to guide future output sentences. Abstracting these inference rules through Cis² would likely allow the language model to better capture and learn CCI.

We also resonate with question-answering based approaches to commonsense inference for stories Lal et al. (2021); Castricato et al. (2022). Lal et al. (2021) trained large language models on their dataset, finding that they only perform well when the answers are present in the narrative. This finding goes hand in hand with our finding that the original GLUCOSE task formulation allows for easy paraphrasing and thus inflated performance.