Exoskeleton for the Mind: Exploring Strategies Against Misinformation with a Metacognitive Agent

Seventeen Japanese participants (13 men, 4 women, no one of another gender identity; aged 22-62) who had used Twitter at least once participated in the research. The research protocol was approved by the university ethics committee (approval #2022170). All participants signed a consent form after receiving an explanation of the research. They were given compensation of equivalent to roughly $11 USD.

We conducted the study online and offline, due to restrictions arising from the COVID-19 pandemic. The offline version was conducted through a computer in the lab. The online version used Zoom and the Quick Assist program available in Windows 10. This allowed the participant to remotely control the lab computer, which allowed us to maintain the same virtual environment online and off, especially in terms of the browser environment.

The pre-screened tweets were fact-checked and categorized as either fake or true. For this, two researchers searched for real tweets about COVID-19 posted on Twitter between March 2020 and October 2021. 26 keywords were used as search terms, e.g., Vaccine, Corona, Corona graphene, Vaccine, Vaccine poison, Vaccine mask, mRNA spike, mask germs, mRNA poison, mRNA vaccine, Corona Electromagnetic Wave, Vaccine Danger, etc. Factual tweets usually referenced source data from the government or research articles, while fake tweets tended to lack evidence or media fact-checking. This led to more manual fact-checking efforts on the misinformation side. Fact-checking involved reviewing government materials and media articles. Tweets that both agreed to be true or fake were selected. From these, 15 factual and 15 fake tweets were randomly selected for the agent’s use.

Qualitative and quantitative data were collected. Participants’ voices and faces during the think-aloud protocol were recorded. Audio was transcribed by a researcher who spoke fluent Japanese. The post-test questionnaire consisted of fifteen items using Likert scales and three open questions. Perceived usefulness (PU) was used as a measure of initial trust (Benbasat and Wang, 2005) and assessed through four items from the instrument by Zhou et al. (Wang et al., 2016) on a 7-point Likert scale. Usability was assessed through the 10-item System Usability Scale (SUS) (Brooke et al., 1996) on a 5-point Likert scale. We created one Likert-style item to assess frequency of Twitter use (no longer a user, never used, used once a month, once a week, daily). The open-ended questions asked about likes, dislikes, and what could be improved about the agent. For the interview, we asked about experience with the agent, use of Twitter, thoughts about Elemi’s design, understanding of the agent’s capabilities, and willingness to use the agent again or not. All items were presented in Japanese; all English scales were translated into Japanese by one of the researchers who spoke fluent Japanese and English, and back-translated by two or more other researchers whose native language was Japanese.

Participants were given a description of the study and asked for consent. Next, they practiced the think-aloud protocol (Ericsson and Simon, 1984). For this, they read cat-related tweets for  5 mins. They then completed the main task using the think-aloud protocol to narrate their experience. For this, they read tweets from a timeline comprising the curated 15 factual and 15 fake tweets, while the agent was activated. Participants were not directed to engage with the agent; they were free to use it or ignore it. Afterwards, they answered a post-test questionnaire and participated in an interview. We then ended the session. Participants were thanked and compensated. The study took about one hour.

We generated descriptive statistics and scored the measures according to the instrument instructions. We used a two-tailed Student’s t-test to compare the usability and PU scores by willingness or unwillingness to use the agent. For the qualitative data, we used reflexive thematic analysis (Braun and Clarke, 2006). An inductive approach was used in the absence of a predefined thematic framework. For this, one researcher read the corpus of data drawn from the think-aloud protocol, interview transcripts, and questionnaires three times. The researcher then developed 41 codes, later categorized under eight high-level codes: (1) experimental environment, (2) user interface, (3) feedback on the agent’s functions, (4) inferences about the agent’s logic, (5) UX, (6) feedback on the content, (7) opinion on the agent’s information, and (8) agent improvements. The researcher then grouped these into three themes, in light of the two RQs. These were reviewed several times with another researcher for clarity and refinement.

Most participants identified the agent’s role as an anti-misinformation tool correctly; however, two did not. These were explainable: a non-frequent Twitter user and a non-student who believed that the timeline itself was created by the agent. We used the data from the fifteen participants who did understand the agent’s role. On average, they opened 43% of the tweets marked as misinformation. The average normalized score for the SUS was 66.91, indicating marginal usability (Bangor et al., 2008). There were no significant differences in usability scores between the eight participants willing to use the agent ($70.25\pm 19.88$) and the seven who were unwilling ($62.14\pm 10.15$). The mean PU score was 4.53 (out of 7). A significant difference between the willing ($5.10\pm 0.65$) and unwilling ($3.71\pm 0.96$) groups, t(15) = 3.576, p ¡ .01, indicated that those willing to use it found it more useful, as expected.

We now turn to a review of our inductive findings from the thematic analysis.

Theme 1: Desire for transparency. Sixteen participants (all but one) wondered about the logic behind the agent. They investigated author profiles, the presence of quotes, the tone of the content, slang use, and other indicators to understand how the agent identified inaccurate tweets. Nine participants found that the filtering was not in line with their criteria and raised it as a problem. P1: ”Well, this tweet hasn’t been deleted, but I honestly don’t believe it.” P8: ”He doesn’t seem like a person who would be likely to publish inaccurate information, so I’m surprised that his tweet is inaccurate, and he’s using foreign data in English, or a summary of a paper, which seems to contain credible information.” Seven participants also expressed concern that the agent was biased and used only official or anti-COVID-19 information. P17: ”I’m a little scared that the information that the agent is letting me see or blocks is decided arbitrarily.”

Theme 2: Trust and reliance. Even when the agent was perceived as mistaken, some participants still trusted its decision. P11: ”Even if the information is correct and the agent is wrong, I’d still accept the agent’s decision.”. In fact, four participants hesitated to trust some tweets, stating that they would follow the agent’s decision without knowing why it was made. P1: ”I may believe this tweet because it’s not hidden by this tool.” P6: ”It’s easy to believe because the agent didn’t hide the tweet.”. Ten responded that they would use the agent in future.

Theme 3: Orientations to visual filtering. Impressions of the visual filtering changed over time with continued use. Nine participants found that the more they used the agent, the more it attracted their attention. P1: ”I want to see what the conspiracy theory is by looking at the hidden tweets.” P4: ”If such radical content is hidden, this might be a cue to click it, so I thought it might be a ploy by someone who wants to spread such radical ideas.”. Five participants also noticed and appreciated that the agent prepared them to consider the content before they opened it. P11: ”It was good to embody the point of stopping and thinking about whether or not it was fake information.”

Still, about half did not necessarily want the agent to take action on misinformation. Eight participants enjoyed reading misinformation, finding it humorous. P13 criticized the filtering: ”It’s fun to hear various opinions, and I gain something from them. We must to decide ourselves whether to accept a specific opinion or not. But this agent takes away such a process. It seems for me there’s no difference between a person deeply immersed in conspiracy theories and users of the agent.”. One participant suggested that it would be better to make the right information stand out rather than hide the wrong information. P3: ”I’d like the agent to comment on something rather than just excluding negative tweets.”.

The agent was generally usable, based on the SUS score (66.91) (Bangor et al., 2008) and qualitative feedback. Frequent Twitter users immediately understood the agent’s purpose and the ”one click” toggle generated trust. Still, the presence of the agent iself drew attention to the tweets containing misinformation, leading many to focus on these fake tweets rather than stay vigilant. This points to the Pandora effect (Hsee and Ruan, 2016): a desire to resolve uncertainty even while knowing that it will have a negative effect. Similar to Pennycook, Cannon, and Rand (Pennycook et al., 2018), the ”fact-checked” explanations provided by the agent served to increase perceived credibility, regardless of factuality. However, the unintentional, negative side-effects need addressing, perhaps with metacognitive scaffolds or a means of allowing the user to guess what the hidden content might be even while not displaying the content directly (Stevens, 2007).

Observations showed participants skipped nearly half of the content containing misinformation. In this sense, the agent was effective at preventing exposure to misinformation in live Twitter. But this experience was not pleasant for everyone. PU was linked to a desire to use the agent in the future. Some participants wished for a greater understanding of how the agent made decisions, while some were satisfied even when they thought that the agent made a mistake. Information that lacks transparency and a clear objective is unacceptable, even when true (Hartwig and Reuter, 2019). Others wished to view misinformation regardless, which may defeat the purpose of hiding it due to the threat of repeated exposure, where familiarity breeds a sense of accuracy, i.e., an illusory truth effect (Hassan and Barber, 2021).

Overall, most people found the surface-level explanations about fact-checking to be insufficient when the agent covered up content. We also needed to tackle the challenge of repeated exposure. We thus returned to the literature, drawing on theories of metacognition as a means of assisting users who wish to engage with content containing misinformation directly. We developed two new versions of the agent, each representing a high-level strategy suggested as promising by previous work: Camouflage (Stevens, 2007), a new form of Cover that blurs out (rather than hides) the content and specifies what parts were misinformed and why, as a metacognitive cue; and Consider, which uses the metacognitive ”pause and reflect” strategy investigated in numerous studies (Fazio, 2020; Bago et al., 2020). We report on the study next.

We recruited people experienced with Twitter (Table 1), with 57 in the first session and 49 completing the second. They were asked about their age and level of education after the second session to avoid stereotype threats (Kray et al., 2002; Mazerolle et al., 2012). Participants were recruited through word-of-mouth, our social networks, and Prolific⁶⁶6https://www.prolific.co/. They were compensated with at roughly $11 USD per session. This study was approved by the university ethics committee (approval #2022170).

The study was conducted online and offline. Participants remotely controlled a lab computer in Zoom. This allowed us to control the study and ensure that all participants experienced the same environment.

The agent used the same fact-checking resources as in the pilot study, but the pre-screened tweets were changed to ensure up-to-date content. 30 factual and 10 fake tweets posted between January and May 2022 were selected. Unlike the pilot study, we allowed participation in either Japanese or English. For this, two researchers translated and back-translated the tweets. Also, all UI text and other content, such as ads and response tweets, were translated in realtime using the Google Translate API. English participants were notified that the original content was in Japanese to avoid confusion due to missed translations or oddities in the machine translation.

Tweet accuracy evaluations, misinformation exposure time, fact-checking activities, and usage logs were collected. Screen recordings with Zoom were used to capture observations of user activities. We verbally prompted participants to judge whether a tweet contained misinformation (Chen et al., 2022). These assessments, as well as exposure time to misinformation content and other user activities, were transcribed from the recordings. The post-questionnaire used the same items as in the pilot study (refer to 3.2.4), in Japanese or English.

There were two sessions, two weeks apart, with a different Twitter timeline each time that included 30 tweets: 20 factual and 10 fake. The 10 factual tweets changed between the first and the second session. To start, participants were given a description of the study and asked for consent. They then carried out the main task, verbally speaking aloud their assessment of each tweet as they went. Unlike in the pilot study, they were asked to read the timeline to the end. Participants were not given any information of the agent and were free to engage with it or ignore it. After the main task, they completed the post-questionnaire, and then were thanked and compensated.

Descriptive statistics were generated. An individual accuracy score was obtained by dividing the number of accurate assessments by the number of tweets (0 to 1), one each per set of factual and fake tweets. We compared the time spent per fake tweets and during fact-checking activities by agent version and session to assess the effectiveness of each strategy and the presence of an illusory truth effect (Pennycook et al., 2018). When groups contained less than five data points, Fisher’s exact test was used instead of a Chi-square test. In addition, we used a Kruskal-Wallis test when comparing the accuracy score ratio, SUS, PU, and the time spent per fake tweet across the three conditions. For the qualitative data, we used an applied thematic analysis approach (Guest et al., 2011) with the goal of generalizing the findings by agent version and across participants. For this, two raters analyzed the data from the three open-ended questions. The primary rater added on new themes and then each rater coded 20% of the data separately. Inter-rater reliability was assessed with Cohen’s kappa, with 0.8 as the criterion for agreement (Landis and Koch, 1977). Themes that did not meet this criterion were discarded or modified. Borderline cases were resolved by discussion. Similar themes were grouped together.

Participants engaged in fact-checking activities (Table 2). For all groups, the most frequent activity in Session 1 was checking ”content attachments,” followed by ”Twitter users reactions” and ”author profiles.” Notably, those in the Consider group engaged at the same frequency in ”content attachment” and ”Twitter users reaction” activities. One-third conducted ”web searches” outside of Twitter. In the second session, the trend was reversed. All in the Camouflage group checked ”Twitter users reactions,” followed by ”content attachments,” ”author profiles,” and ”web searching.” All in the Consider group checked ”content attachments,” followed by ”Twitter users reactions,” ”author profiles,” and ”web searching.” All in the agent conditions performed at least one fact-checking activity in the second session. Regardless of condition, regular Twitter users actively sought out additional information before they judged whether the content was factual or fake.

Accuracy scores for fake and factual tweets alongside exposure time to fake tweets per session and agent version are in Table 4. In the Camouflage group, seven (of 19) identified misinformation in the first session and twelve (of 17) did so in the second session. These scores were compared by agent version and across time. No statistical significance was found for accuracy score or exposure time.

Agent engagement frequencies are in Table 3. In the first session, seven (of 19) participants opened the Camouflage agent’s filter and read all false tweets. In the second session, twelve (of 17) opened up the filter and read the content at least once, and eight (of 17) uncovered nearly all content. In the Consider group, one (of 19) interacted with the agent in the first session. Fourteen (of 17) in the second session used the agent at least once, and seven (of 17) interacted with it continuously. In the first session, the Camouflage version of Elemi was interacted with more than the Consider version (p = .042), but not in the second session (p = .688).

More than half of participants in Session 1 did not interact with Elemi As such, only the scores for Session 2 were compared. The SUS for Camouflage was $5.24\pm 1.14$ and for Consider was $4.37\pm 0.79$. There was a statistically significant difference found between Camouflage and Consider, t(32) = 2.621, p ¡ .05, favouring Camouflage. For PU, Camouflage was $68.3\pm 14.4$ and Consider was $53.8\pm 11.4$. There was also a statistically significant difference found, t(32) = 3.250, p ¡ .01, indicating Camouflage was preferred.

Use of Elemi varied by version, i.e., the metacognitive strategy it deployed. When it came to the Camouflage version of Elemi, two participants were not aware of its existence. According to P40, ”I wasn’t even aware of the presence of an agent.” Among those who used Elemi, one (P18) expressed concern about dependence on the agent’s judgments when in Camouflage form: ”I tend to judge by the content of the agent and less by the content of the tweets.” In contrast, the Consider version was intentionally not used by five participants. According to P50: ”I dislike it because it just provides opinions, there’s little objective about it.”

Participants had various reactions to the user friendliness of the agent forms. Nine users of the Camouflage version expressed that the agent was easy to use. As per P54: ”It’s lightweight and easy to learn.” Regarding its transparency and explainability, two user who never opened up its filter expressed positive opinions about its explainability. According to P41: ”It’s easy to use and integrated well with the social media platform.” Still, eight (of 12) who engaged with the agent expressed suspicion about its transparency and explainability. For instance, P13 wrote: ”I doubt the authenticity of the agent’s judgment …” Another advised that the Camouflage version of Elemi may be a fit with other platforms, especially educational ones: ”I think that detecting lies is a necessary skill when you grow up, so I feel it would be good to make it for children” (P26). For two Consider users, the agent was friendly. As per P8: ”The design is catchy and friendly.” But when it came to ease-of-use, eleven (of 14) participants noted that the agent was not so easy to use. They all criticized the reasoning process. As P11 described: ”I dislike having to go through each step …”

Ten participants spoke to the Camouflage version of Elemi. While six said that the accuracy of the fact-checking was high, the other four indicated that some of the agent’s judgments were incorrect. According to P16: ”I feel that the sources for judging misinformation were somewhat scarce …” In short, more fact-checking was desired. At the same time, no comments were made about the Consider version of the agent or other aspects of its performance.

Both versions of the agent were deemed good for decision-making support, i.e., as metacognitive tools. Still, the perceptions of Elemi’s capability differed by version. Participants viewed the Camouflage version of Elemi as an agent that confirmed the correctness or incorrectness of information. In contrast, Consider was characterized as an agent that encouraged a rational decision: a different method of assistance. For Camouflage, six participants commented that making incorrect information invisible would lead to platform comfort. According to P2: ”The collection of tweets with credible sources was pleasant to see.”. Two said that the Camouflage version of Elemi led them to be more aware of fake news. As per P33: ”The warning made me more alert.”. Two others said that, on the contrary, their interest in the content of fake news rose. P31 explained: ”Using an agent arouses interest in the fake news it’s hiding.” Five Consider participants said that the agent inspired them to think logically. According to P7: ”It helped me to judge logically rather than intuitively.” However, seven participants felt that there was a lack of objectivity in the agent’s decisions. As per P52: ”It’s mainly or only opinion, there’s little objective about it.” For P18, the Consider version made it difficult to use the platform as casually as usual because it encouraged them to think about the information: …”I agree that fact-checking is important, but I felt that the value of Twitter as entertainment decreased …”

Leaving engagement with the agent up to the participant led to low engagement in the first session. Still, the accuracy score ratio of the Consider version of the agent did not increase in the second session. As such, this strategy did not lead people to deliberate on the content as described by Bago, Rand, and Pennycook (Bago et al., 2020). As for Camouflage, while it worked in the first session, its efficacy was disrupted in the second, apparently when tweets were repeated over time, i.e., repeated exposure, in terms of both accuracy score and user doubt. Transparency and clear explanations about the agent’s decisions (Hartwig and Reuter, 2019), such as misinformation keywords and fact-checking references, did not prevent perceptions of the performance of the Camouflage version of the agent from degrading on re-exposure to the same misinformation.

A phased, time-based approach to assessing the agent revealed the weakness of the Camouflage version, indicating that it would not be superior to the Consider strategy in the wild, at least in terms of helping people distinguish between factual and misinformed tweets to which they were exposed multiple times. Indeed, neither Camouflage nor Consider alone were ideal as an anti-misinformation strategy over time. Camouflage, which provided fact-checking data from the agent, was increasingly questioned by users from session to session. Given that agent behaviour can make users skeptical (Chen et al., 2022), repeated usage could increase the chance of users encountering inconsistencies with agent decision-making. These inconsistencies could exacerbate perceptions of distrust towards agent performance. Consider, which required cognitive effort and made people think for themselves, was described as hard to use and inconvenient, having marginal acceptability and decreasing the entertainment value of social media. Consider users also wanted objective information to support their decision-making rather than engaging in reasoning activities while reading the content. In the absence of objective feedback, they rated the agent’s usefulness as poor, equivalent to the level of the ”will not use” group in the pilot study. In Twitter, where content is instantaneous and unchallenged (Starbird et al., 2010), taking on cognitive load was opposed by participants. Therefore, a more practical approach could be to combine strategies, harmonizing the two agent versions as one and potentially including other strategies that we did not assess.

Changes in participants behaviour were captured over the course of two sessions. Unlike in previous research (Pennycook et al., 2018; Hassan and Barber, 2021), the illusory truth effect was not found. Still, a notable difference was found in the Camouflage group. Although it did not reach statistical significance as in Chen et al. (Chen et al., 2022), the Camouflage group tended to be more skeptical than those in the Consider and control groups over time. The metacognitive cue that Camouflage used, blurring certain tweets as an indicator of misinformation and thereby implying that the rest was factual, was questioned by participants. The more inconsistent it was felt to be over time, the more that participants felt distrust towards the agent. Some even felt that the agent was misguiding them on purpose. The number of participants who disagreed with the agent doubled from the first to the second session (2 to 4). We might call this an amplified disbelief effect. In short, those who had doubts about the agent’s performance the first time were inclined to feel twice as doubtful the second time around. Thus, there is a chance that those who mistrust an agent from the start will further mistrust the agent in subsequent sessions.