Portrayal: Leveraging NLP and Visualization for Analyzing Fictional Characters

We identified several common workflows for both writers and scholars.

During the brainstorming phase of the interviews, several participants were at first confused about how characters can be represented computationally and how that can help them. Screenshots of existing visualizations (Figure 2) helped them understand our research proposal. Writers speculated that such a tool for visualizing character traits could help them identify both intentional and unintentional characterizations. They thought such a tool would be most useful during revising when reorganizing their thoughts and trying to find a better line of delivery. Scholars noted that such a tool could work as a visual index and help them analyze character traits more thoroughly. Participants further provided requirements and suggestions for such a tool, which are discussed next.

This section outlines the system requirements identified from the interviews.

To support R1-R6 and DG1, we first need to detect all reference to the characters in the full text. For example, consider the following two sentences: Gary is a journalist. He is a writer too. Here “he” refers to “Gary.” Identifying such co-references and clustering them correctly for a story is challenging as book length coreference resolution is an open problem in NLP (Han et al., 2021). However, this is critical for our case as extraction of traits such as actions and emotions will depend on correct identification of where the characters appear in the text.

To address this challenge, we tested three popular co-reference models: NeuralCoref (Face, 2021), the AllenNLP coreference model (Lee et al., 2018), and the bookNLP co-reference model (Bamman et al., 2020). To evaluate the models, we split the books in our test dataset into chapters and then ran the models in each chapter to detect the co-reference clusters. The chapter-wise split provides a manageable way to validate the coreference clusters. We then calculated Error rate, the average number of errors in the detected clusters for each model. An error occurs when a mention has been incorrectly assigned to a cluster. Overall, AllenNLP model had the lowest error rate, 1.5 (SD = 0.85). NeuralCoref and bookNLP had on average 5 (SD = 2.1) and 4.1 (SD = 1) errors, respectively. AllenNLP also detected on average 2 more clusters than the other two models. Thus, we decided to use AllenNLP for our system.

While AllenNLP is the most efficient among the three models, we noticed some mistakes by the AllenNLP model. Such errors can lead to different traits being assigned to wrong characters. AllenNLP also does not assign any name to a detected cluster. Considering these factors, we adopt a human-centered approach to detect and validate the characters in the full text. We used an interactive interface to validate, name, and merge the clusters (Figure 3). Finally, note that the coreference resolution already captures all mentions (i.e., presence) of each character, addressing R1.

To detect the actions, we need to identify verb and character pairs where characters are agents, performing the actions specified by the verbs. Open Information Extraction (Open IE) (Stanovsky et al., 2018) matches this requirement (R2). Table 3 shows an example sentence and two propositions extracted by Open IE. We identify the agent (ARG0) and verb (V) pairs and search for the agent (ARG0) in coreference clusters for a possible match with a character. In the case of a positive match, we assign the verb to that character. We manually annotated the actions for the lead characters in our test dataset and calculated the accuracy of this action detection method. It yielded a 95.7% accuracy.

To facilitate speech analysis, we extracted all direct quotes using a rule-based approach. For each direct quote, we look for self-referring words such as “I”, “me”, and “mine” and match with the coreference clusters. In case of a match, we assign the character as the speaker of the speech. For the speeches with no such match, we look for the closest verb just preceding or following the direct quote in the same sentence. Here, we use part-of-speech tagging to determine if a given word is a verb. Next, we find the corresponding subject for the closest verb using dependency parsing. Here, the subject can be a noun or a pronoun. In case of a pronoun, we use coreference resolution to determine the corresponding noun. This noun is attributed as the speaker of the direct quote. To evaluate this method, we again manually annotated the direct speeches or discourses from the lead characters in the test dataset. This method yielded a 98% accuracy on our test dataset.

Adjectives work as direct definition to characters (R4). To detect direct definitions, we first iterate through all words and identify adjectives using part-of-speech tagging. Next, we try to identify the corresponding subject for each adjective. To do that, we use dependency parsing and hop through different nodes in the tree from the adjective to a noun/pronoun. This is achieved in two steps: (1) moving from the adjective to its parent node until we find a verb; and (2) moving from the verb to its subject. In case any of these steps do succeed, i.e., we do not find a corresponding verb or subject, we ignore that adjective and move on. In case the subject is a pronoun, we use coreference resolution to find its corresponding noun. This entire process is repeated for all adjectives to find a mapping between an adjective and its subject. Lastly, we aggregate all adjectives corresponding to each character. This method was 91.2% accurate on detecting adjectives for corresponding characters on our test dataset.

We used sentiment analysis model from Allen NLP (90% accuracy) to detect sentiment for each sentence (R5). It predicts sentiment score on a binary continuous scale (-1 to +1).

We also used two pre-trained models for emotion detection. The first model is Google’s T5 base model (Raffel et al., 2020) fine tuned on a emotion detection dataset (Saravia et al., 2018). It detects emotion on six categories: joy, sadness, love, anger, fear, and surprise. The second model is a RoBERTa base model (Liu et al., 2019) fine-tuned on the above emotion dataset. Users have the choice of selecting any of the two models to use.

To extract contextual and environmental information, we used both coreference resolution and entity detection (R6, R10). We noticed that, aside from characters, coreference resolution can also detect entities such as places and events with repeated mentions. Using the annotation interface (Figure 3), we can label a cluster as a contextual tag (Figure 3). We further ran Spacy’s entity recognition model to detect any missing places or time references. Finally, writers often add small titles to the chapters that can work as contextual information. We extract those using regular expression.

While action is an important trait, our formative study suggests writers and scholars also value changes in actions over time (R8). To calculate change, we built upon Relative Norm Difference (Garg et al., 2018), a metric that can measure difference between two sets of embedding vectors. Let $A_{ic}$ be the set of actions for character $c$ from chapter $i$ and $A_{(i-1)c}$ be the set of actions for character $c$ from chapter $i-1$. Let $m_{ic}$ and $m_{(i-1)c}$ be the mean vectors for all the words in $A_{ic}$ and $A_{(i-1)c}$. We define the change between $A_{ic}$ and $A_{(i-1)c}$ as $dist(A_{ic},A_{(i-1)c})=cosine(m_{ic},m_{(i-1)c})$. The range for this function is 0 to 1, where 0 means no change and 1 means the highest amount of change.

Change is also important for sentiment (R8). However, sentiment may fluctuate significantly on short text intervals (e.g., in a passage), making it difficult to see long-term patterns. Since sentiment is a continuous variable (between -1 and +1), we used moving average to smooth out short-term fluctuations and highlight longer-term trends and changes. We used $n=5$ as window size for a full length story and $n=3$ for a short story.

To support DG2, we visualized each trait indicator as a separate card, following the design of the popular UI component with the same name.³³3https://www.nngroup.com/articles/cards-component/ All card views follow the same design. In that way, a card works as a modular and homogeneous unit for our interface. For example, Figure 4 shows the basic components of a card: the name of the indicator, information about the indicator, control and filtering options for the indicator, a cancellation icon, and the main visual representation.

Motivated by the dispersion plot commonly used in the humanities (Jacobs, 2018; Hoque et al., 2022), we used an occurrence matrix to represent trait indicators that depend on time (R11, DG2). Figure 4 shows an example of occurrence matrix-based timeline. By default, the horizontal axis represents chapters in the story. If there are no chapters, the axis defaults to one chapter only. The vertical axis represents the characters.

We encode indicator values using a color scale. For example, in Figure 4 (top row), we encode the number of mentions for the characters in each chapter in the matrix cells with a linear color scale (i.e., the brighter the color of a cell, the more are the mentions for the character in the specific chapter). Similarly, in Figure 4 (bottom row), we encode the number of direct speeches for the characters. Figure 1 shows another example where three timelines—one for each trait indicators direct discourse or speech, sentiment, and emotion—are aligned vertically.

There are two ways to interact with the timelines: (a) mouse hover and (b) mouse click. These interactions are designed to link the visualization with the text (DG4). On hovering over a cell, we show the character and chapter name in a popup. We also highlight the characters that co-occur with the selected character in the chapter. This mechanism extends across multiple cards. For example, in Figure 4, we hover over a cell for the character Mr. Elliot (shorthand Mr. E) in the presence card (from Jane Austen’s Persuasion (1818)). We notice that there is no corresponding cell for Mr. Elliot in the direct discourse or speech view. This suggests even though Mr. Elliot is highly mentioned in the chapter, he did not have any direct discourse.

To investigate further, a user can click on the cell. In that case, the cards show the sentences of the chapter (Figure 5). Interestingly, we see that there is a lot of direct discourses for Anne Elliot (shorthand Anne E.) and Mrs. Smith (Mrs. S.), but none for Mr. Elliot. The explanation lies with the fact that Anne Elliot and Mrs. Smith are conversing about Mr. Elliot’s past affairs and intentions. Figure 5b shows a long speech from Mrs. Smith where she talks about Mr. Elliot’s lust for money. Note the colored background for the “I” in the first sentence which indicates the speaker and why our model thinks Mrs. Smith is the speaker in this case.

Four trait indicators (presence, direct discourse, sentiment, and emotion) in our interface use the occurrence matrix representation. While we wanted to keep a homogeneous design across all visualizations, some trait indicators were not suitable for visualizing in a timeline. More specifically, actions and direct definitions are mainly words and do not have a specific score attached to lay them out in a timeline. Therefore, we visualize them in word zones (Hearst et al., 2019). Figure 6 shows a word zone for the character Zora from the story The Quest of the Silver Fleece (1911) by W.E.B. Du Bois. To cluster the words into semantic groups, we perform $k$-means clustering on the embedding of the words. Further, we identify the font weight for a word ($w$) for a character ($c$) as: $weight(w,c)=tf(w,c)*(1/df(w))$, where $tf(w,c)$ is the frequency of $w$ for $c$ and $df(w)$ is the frequency of $w$ in the whole story. It is essentially a normalized version of tf-idf.

Actions are also visualized in the same way. However, based on DG3, we also wanted to visualize changes in the actions. Since this a time-dependent indicator, we again use the matrix view to visualize it. Figure 7 shows this representation for the story Persuasion. While we retain a similar representation, the interaction behaviour is slightly different for this indicator. Because the number of actions can be large for a chapter, we rank and summarize the actions in a popup (Figure 7b).

We show the contexts in a separate optional card (DG5). The contexts align with the chapters in the timeline. For example, Figure 7 shows the contexts on top of the changes in action card. We also highlight the contexts when a user interacts with the timelines. For space restriction and aligning the contexts with the cells, we can only show a limited number of contexts. To avoid overlaps among them, we use dynamic programming to find vertical positions of the texts whereas the horizontal positions are determined by the corresponding chapter.

We decided a qualitative study is best suited to evaluate Portrayal for three reasons. First, creative works such as fiction writing and analysis typically do not adhere to predefined structures and largely depend on artists’ styles and idiosyncrasies. Thus, there is no objective measurement of efficiency for such tasks, which quantitative studies often try to measure. Secondly, we anticipated that writers and scholars will not be comfortable if we put them into analytic tasks in a controlled environment for a limited time. Thus, we wanted participants to use the tool in a familiar environment for an extended period, with the freedom to adjust to the tool at their will. Finally, there are currently no analytic tools that focus on characterization against which Portrayal can be quantitatively compared. Although our work is built upon several prior works (Section 3.3), none of them were designed to understand characterization. A logical baseline is a simple text editor without any analytic support. However, given that our participants are professional writers and scholars, we believe they are already in a position to compare Portrayal with a simple text editor.

Aside from the guiding multiple choice questions (Section 7.2), we did not provide any explicit task list to the participants but instead encouraged them to follow their intuition and own creative work process. This decision is again motivated by allowing participants to practice their artistic freedom at will. Another critical decision is whether or not we should ask writers to write a story using our tool. Writing a narrative fiction takes significant planning and effort, and is likely to be a very long and involved process. We would require long time commitments from several writers. We felt that this was impractical at this stage of the research project. Thus, we instead decided to ask writers to analyze one of their work-in-progress drafts. This allowed us to understand what feedback Portrayal can provide to writers during revising, one of the critical stages of writing (Du et al., 2022b). Additionally, writers provided feedback on how they would use the tool during writing from their personal experience.

Writers in our study did not report using any analytical support before Portrayal. They primarily relied on critical reading, multiple rounds of editing, and obtaining feedback from peers, friends, and publishers to revise their writing. With the first use of any analytical support, participants shared several insights they gathered from Portrayal, some reassuring, while others were surprising. Figure 8 presents some of these insights. We further present the themes that emerged from the post-study interviews below.

Creating Dynamic Characters and Scenes. A critical challenge for writers is to create dynamic characters and scenes that engage readers. Participants thought the indicators supported by our tool helped them in this regard (P5-12), validating our first design goal (DG1). We also noted that the use of multiple views helped writers analyze multiple indicators together (DG2). For instance, P10 mentioned that sentiment and emotion are helpful for creating characters that go through many emotional changes.

‘‘I think sentiment and emotion are particularly useful for characterization. A good rule for characterization that I follow is having multifaceted characters that can basically show a variety of emotions and feelings. I want to have dynamic characters which are at some point optimistic but at other point is pessimistic, unless I want to have static characters. Using the tool, I can see parts where I want to make changes to achieve that goal.’’ (P10)

Sentiment and emotions are also helpful for creating scenes where characters with opposite emotions interact. In P11’s opinion, such scenes create “tension” and make the story more “engaging”.

‘‘One thing I try to follow, not that I achieve that all the time, is to have characters with different emotions interact. For example, interactions between a joyful character and a sad one may help create tensions in the story. The tool helped me validate that in my story.’’ (P11)

Another set of participants found presence and direct speech to be most helpful for creating dynamic characters and scenes. For example, P8 mentioned that the tool helped them validate Bakhtin’s Polyphonic Theory.

‘‘If you want to create a polyphonic novel, several characters should have voices instead of just one. The alignment of timelines (in presence) is really helpful to see whether you have enough characters in the scenes and chapters. Discourse (speech) analysis enhances that by showing how you manage the voices of the characters.’’ (P8)

Finally, actions and direct definitions are helpful for determining whether a character has developed or not. For example, Figure 8c shows one such example where the lack of direct definition helped P10 identify an underdeveloped character.

Capturing Character Arcs. Participants found that Portrayal was able to capture character arcs (P5, P9, P12). A character arc is a transformation or inner journey of a character over the course of the story. Changes in action and sentiment can partially capture it, thus validating DG3. For example, P5 and P9 said:

‘‘I think the change in action is one of the most useful for sure. Just being able to see the way actions jump from chapter to chapter is super interesting. It would be really useful for thinking about a character arc.’’ (P5)

‘‘The smooth option in the sentiment view was awesome. It perfectly captured the arc for my lead character!’’ (P9)

Identifying unconscious bias. We noticed Portrayal helped writers expose their unconscious bias towards a group of characters with a similar social identity (P6-7, P9, P11). Writing a story, even a short one, takes significant effort and often requires multiple rounds of revising. While reviewing a draft, Portrayal can be helpful to reassure intended and unintended characterization for a group of characters. Participants found the presence and discourse indicators to be most beneficial for that. We note that this finding is in line with the findings of DramatVis Personae (Hoque et al., 2022). For example, P9 and P6 said:

‘‘A creative writer gets an objective perspective on which characters they give more presence and dialogue to in a story. This could help writers address their own subconscious biases as they could see it laid out for them---what type of characters they give priority to and what characters they let interact.’’ (P9)

‘‘I want to see whether the female characters in my novel have had enough space for expression. But when I am writing, I am not consciously thinking about all of this. I am just writing and writing and writing. And then, when I review what I have written, I can see whether my female characters have the right exposure.’’ (P6)

All scholars in our study reported close reading as the primary way to analyze a literary work. Four scholars (P2, P4, P9, P12) reported having hands-on experience with text mining and NLP for distant reading. However, none reported using any special purpose tool for close reading or analyzing characters. We report how Portrayal augmented literary analysis below, based on the themes that emerged from the interviews.

Balancing Subjectivity and Objectivity. The indicators in Portrayal will help scholars to support their literary argument with concrete examples and avoid unconscious predilection toward an argument (P1-3, P9) (DG1). Scholars typically support their argument with a limited number of examples, which is susceptible to missing out on examples that oppose their argument. For example, P2 and P9 said:

‘‘This tool will help scholars such as myself balance our subjectivity with the objective information provided in the tool. If I did not have this tool, I am only analyzing maybe three actions of a character. But, there may be one hundred and eighty-seven actions that do not support my argument, and I can unconsciously turn a blind eye to them. The tool removes this kind of blind subjectivity for a scholar and makes you look at the text of what the author put forward.’’ (P2)

‘‘It has been very challenging since the beginning for literary analysis that we say this character is too pessimistic. But how do we support it? We hugely rely on sampling. We say in these three examples, he shows negative emotions. But here, you have more concrete evidence for the claims that you make.’’ (P9)

On the flip side, participants suggested that it is possible that scholars can get over-dependent on the tool and lose their subjective intuition (P1-2). However, participants suggested this is unlikely since scholars are trained to exercise their literary knowledge and will likely find a balance between subjectivity and objectivity.

Bridging Close Reading with Computational Support. Portrayal provides a bird’s eye view over a story (P3) and helps scholars drill down in a top-down approach for close reading (P1-P4, P10-12). Close reading is an integral part of literary analysis. However, participants mentioned that it requires enormous effort and time to annotate text. The links between text and visualization reduce workload for scholars (DG4). P10 and P12 said:

‘‘Most digital humanities tools (e.g., Voyant Tools) are designed for big data (distant reading) But this one helps with the close readings because it breaks down every character, every chapter, and even every sentence. Moreover, it starts with an overview of the chapters and helps me drill down the chapters of interest. This was a completely new and amazing experience for me.’’ (P10)

‘‘The best feature is the micro level link between visualization and text. I think I will be blind without it!’’ (P12)

Facilitating Literary Conversation and Debates. Participants thought subjective indicators such as sentiment and emotion that depend on the interpretation of the AI model are more suitable for literary conversation and debates (P2-4, P9). During the study, several participants (P2-4, P9) organized the indicators into two categories: (1) Objective indicators such as presence, speech, and actions that are directly present in the text; and (2) Subjective indicators such as sentiment and emotion that depend on the AI model’s interpretation.

Participants suggested that the amount of agreement and disagreement with the subjective indicators will vary from scholar to scholar. This is in line with the current practices in literature where scholars can interpret the same text in different ways. This process makes these indicators a good candidate for literary conversation and debates.

‘‘Even in literary classes, when we used to do analysis, we had debates over what is the sentiment here, what is the emotion this character is experiencing. The tool could be a jumping-off point where you could say like, why would the tool pick a negative sentiment here? Let’s talk about keywords here.’’ (P3)

‘‘It is interesting to see how the AI is inferring sentiment. I think this could be a talking point for people who want to use technology to analyze stories. You can find some controversial examples from the tool.’’ (P4)

P4 further mentioned that such debates and conversation are generally constructive. However, it is possible that scholars who use Portrayal might receive criticism, as the use of AI is a polarized topic in the scholarly community.

Writing Literary Essays with Portrayal. All scholars in our study had already graduated from college. However, several participants (P1-4, P10, P12) reminisced about their college days during their studies when they had been writing literary and term final essays. These participants felt that the tool would be useful for that. For example, P1 said:

‘‘When I am trying to write an essay, what’s usually the hardest thing to do is find like oh when did the character say this or did that, or when did this character show up or find passages from the book and connect them. Being able to do this with the tool will honestly be life-changing for students.’’ (P1)

All participants admired the visual design of the interface. Participants repeatedly used praises such as “cool”, “neat”, “not overwhelming”, “fluid”, “easy to use”, and “easy to get going” to describe the system’s usability. This overall positive experience was reflected in the post-study usability ratings provided by the participants (Figure 9). Across all system usability categories, there were no ratings less than 5 (Figure 9a). Similarly, participants rated most trait indicators highly in terms of how useful they are (Figure 9b).

One usability issue was the lack of a timeline for the word clouds (direct definition and actions). For full length stories, P6 found word clouds to be difficult to use for finding patterns. P6 suggested organizing the words in a timeline for better interpretability. Another usability issue was the long list of actions in the “Changes in Actions” view (Figure 7b). P1 found it difficult and time-consuming to read through the list. P1’s negative experience was reflected on the rating of 1 for this view (Figure 9b-second row). P9 found the color scheme in the Emotion view difficult to understand and wished for a smooth option similar to the one we have for Sentiment.

Finally, participants reported that they frequently used pairs of indicators using our multiple view design. However, use of three indicators was rare, slightly diminishing the effectiveness of DG2. The increased attention required to analyze three indicators together have discouraged users in such scenarios.

While the experience with the tool was mostly positive for the participants, there were a few concerns and limitations outlined by the participants. We discuss them below.

Lack of explanation for the AI models. Several participants inquired about the underlying models that were used in Portrayal (P2, P6-8, P12). During the study, we provided participants with a description of the models and links to the original models. However, most of our models were based on deep learning, making them black boxes with very little explanation of how they work internally. This clearly created a lack of trust among some participants, especially for the subjective indicators (sentiment and emotion). We discuss ways to mitigate this concern in Section 8.

Errors in NLP Pipeline. Participants found a few errors with the co-reference resolution model. It was expected since these models are not perfect. However, participants suggested that it did not impact their experience negatively as they were surprised with the accuracy of the co-reference model, especially in complex sentences. Or, as P1 put it: ‘‘The tool could determine mentions that I might have missed when close reading.’’ (P1)

Additionally, participants found a few adjectives that were wrongly determined as the direct definition of the characters. Due to some ambiguous cases, our rule-based model could not discern and can be resolved easily with additional rules.

Participants requested a few new features in the tool. First, P9 noted that comparing trait indicators in a corpus would be interesting for facilitating distant reading. Similarly, P2 and P12 requested features that can compare two versions of the same story to compare their revised version with the original one. P11 suggested including “point of view” as a feature. For example, writers often narrate the story from the point of view of a narrator or multiple characters. Finally, P6 requested the ability to show conversations more clearly. We include all speech of the characters in the direct discourse view. However, we do not identify or show who is spoken to in the speech, only who is speaking. This information can show how writers manage conversations more clearly.