The Detection and Understanding of Fictional Discourse

Written fiction continues to play an important role in the lives of readers. Despite predictions about the end of the book Phillips (2019) or the death of the novel Aravamudan (2011), fiction remains a central medium of communication that contributes to a sense of meaning, joy and imagination on the part of readers over their life-spans, from childhood to old age.

Written fiction can take many forms. It can be found within a multi-billion dollar industry of major publishing houses (“big fiction,” Sinykin (2023)), self-published books by amateur writers, vibrant online communities devoted to fan fiction, independent presses seeking to promote more diverse styles and voices, or major digital cultural heritage archives.

The computational detection of fictional discourse – identifying whether a text is telling an imaginary story versus a true one – can be a useful task for two reasons. First, it can help locate fiction within large, unmarked digitized cultural heritage collections, thus enriching our knowledge about the past Underwood et al. (2020); Bagga and Piper (2022); Hamilton and Piper (2023).

Second, the use of predictive modeling can also facilitate the identification of distinctive qualities of fictional discourse thereby highlighting its potential value to readers and society. While prior theoretical work has made strong claims about the absence of distinctiveness surrounding fictional discourse Searle (1975); Currie (1990), computational research in this area has shown that fictional discourse represents a very coherent and historically consistent set of linguistic practices Underwood (2014, 2019); Piper (2016).

In this paper, we seek to further contribute to knowledge about the detectability and understanding of fictional discourse. Prior work has focused on using lexical features (i.e. bag-of-words) Underwood (2014, 2019) and LIWC features Piper (2016) to detect fictional documents within large and small collections of historical texts respectively. In this paper, we expand the number of data sets used to test the accuracy of our models beyond prior work. In addition to using historical collections derived from the Hathi Trust Digital Library Bagga and Piper (2022), we also include data from contemporary professional publishing Piper (2022), fan fiction, social media Ouyang and McKeown (2015), historical folk tales from around the world from Project Gutenberg, along with a collection of contemporary anglophone fiction from non-Western sources.

Second, where prior work has used either lexical features or LIWC’s psycho-linguistic categories to represent model features, here we rely on word “supersenses” to capture more general semantic categories of words (see Table 1). Supersenses are derived from Wordnet’s taxonomies and generated through the latest model of bookNLP Bamman (2021). We find that word supersenses serve two important functions for our analysis.

First, they not only help generalize our understanding of the semantic behavior of fictional discourse beyond individual keywords. Additionally, they help operationalize our understanding of fictional writing as a distinctive form of “discourse,” which we define as a socially constructed form of communication Halliday and Matthiessen (2013); Taylor (2013); Berger and Luckmann (2023). We assume that fictional discourse is strongly shaped by the distinctiveness of its agents, actions, and worlds, all of which have a strong semantic aspect (though there may be other kinds of distinguishing features that could be the subject of future work).

Finally, we also explore prediction at the sentence-level using a fine-tuned BERT model to better understand the minimal tokens necessary to distinguish fictional discourse.

In the sections that follow we describe the principal data sets used along with the experimental set-ups employed to better understand both the predictability of fictional discourse and its distinctive qualities as seen from a general semantic viewpoint. We imagine future work could continue to compare these results across more diverse data sets as well as test higher-level formal features.

Table 2 provides a snapshot of our document-level classification experiments with the full results reported in the Appendix. In Table 3 we provide an overview of our sentence-level classification results.

This paper has shown that much of the earlier findings around fictional discourse’s predictability and its emphasis on embodied behavior hold true across a diverse array of corpora. As we discuss in our limitations, future work will want to continue to expand the diversity of genres, languages, and cultures as well move to more non-semantic features to further deepen our understanding of the role that fictional storytelling plays around the world.

One of the core limitations of our work is the application to English-language texts. While we have expanded the range of text-types, historical periods, and regional cultures used in fictional discourse analysis compared to prior work, future work on multilingual analysis awaits.

Our insights also only pertain to the semantic dimension of fictional discourse. While we find the use of “supersenses” a valuable tool for generalizing about semantic behavior, future work will want to focus on the unique formal or structural qualities of fictional discourse when compared to non-fictional narratives.