Fact-checking AI-generated news reports: Can LLMs catch their own lies?

Large Language Models (LLMs) have revolutionized the field of Natural Language Processing (NLP), effortlessly performing tasks that were traditionally considered highly challenging. Their performance is particularly impressive in generating natural language text. Models like GPT-4 can generate coherent, fluent summaries, accurately translate text between languages (especially those with a strong online presence and ample training data), and refine human writing to enhance fluency and appropriateness in tone and style for specific purposes. This technology has the potential to significantly increase productivity across many industries, offering endless applications. However, with this potential also come risks if they are not used properly. One of the main risks is that they can be easily used to generate convincing and yet factually incorrect text, either intentionally or unintentionally. For example, with a simple prompt like “Generate a news report about volcano eruption in Massachusetts, USA”, GPT-4 can generate a news report starting with this first paragraph:

”Massachusetts, USA – May 29, 2024 – In an unprecedented and shocking event, a volcanic eruption has occurred in the state of Massachusetts, an area not typically associated with volcanic activity. The eruption took place early this morning in the central part of the state, near the town of Worcester, sending residents and scientists alike into a state of disbelief and concern.”

Although there has never been a volcanic eruption in reality, the news report is coherent and fluent. Coupled with modern media platforms, such LLM-generated content can quickly spread and reach a large audience. An example is the emergence of AI “news” farms that produce news reports with LLMs to generate advertising revenue with little concern for their impact on society Puccetti et al. (2024). The machine-generated reports can cause confusion and chaos and disrupt the proper functioning of the society. In fact, studies show that false news tends to spread “farther, faster, deeper” than true news, as it often contains novel content that people are more likely to share Vosoughi et al. (2018).

In this study, we present experimental results to answer the questions of whether LLMs are capable of telling if the news stories they generate are truthful and how well they can catch factually incorrect claims in those news stories. We generated 92 news stories with a simple prompt such as ”Write a story about Kobe Bryant rejoining the Lakers” with two LLMs, GPT-4o OpenAI et al. (2024) and GLM Du et al. (2022), all stories with some incorrect claims. These false stories vary by how untruthful they are. Some stories report events that are simply impossible, such as a story about Kobe Byrant rejoining the Lakers, as the former Lakers star has passed away. Some stories report events that are not out of the realm of possibilities but are highly unlikely, such as a volcano eruption in Massachusetts, as the area is not known for volcanic activities. Other stories are about events that have actually happened or are scheduled to happen, but with the wrong time, location, or participants.

We performed our experiments in two settings. In the first setting, we simply provided the full story to GPT-4o and GLM as input and asked if they are truthful. In the second setting, we manually decompose each story into individual checkable atomic claims. A checkable claim can be either an event with specific participants, location, or time, or they can be a state (e.g., Massachusetts borders New Hampshire) or recurring event that holds for an extended period of time such that the exact time is irrelevant. We perform manual “decontexutalization” Choi et al. (2021) on these checkable claims so that they can be verified outside of the context of their document. For this setting, we also experimented with using these checkable claims as queries, providing the results retrieved via the Google Search Serper Api ¹¹1https://serper.dev/ to GPT-4 to assist in evaluating the veracity of the claims within a Retrieval-Augmented Generation (RAG) framework Lewis et al. (2020).

The results of our experiments show that GPT-4o and GLM are very good at detecting stories that contain incorrect claims (and all of them do) when they involve well-known entities (e.g., Kobe Bryant rejoining the Lakers), but they are quite uncertain about recent events that are unlikely. At the level of atomic claims, a significant proportion of them are incorrectly assessed: either a factually correct claim is judged to be wrong or a factually wrong claim is identified as being correct. For an even larger proportion of atomic claims, the LLMs simply cannot decide. When provided with results retrieved via the Google Search Serper Api, the number of non-assessments decreases significantly, accompanied by an increase in both correct and incorrect assessments. Interestingly, even when the Google Search Serper Api returns no results for a claim, GPT-4 still attempts to provide an assessment instead of declining to answer. It appears that simply knowing no results were retrieved is enough to prompt GPT-4 to make a guess. Even with RAG, there is still a significant proportion of claims that the LLM cannot provide an assessment. This means that any solution to fact-checking machine-generated news reports needs to include functionalities on checking claims about new event occurrences that are not checkable against existing knowledge sources. While there has been recent research that shows the promise of using external resources or tools to improve the factuality of LLMs Gou et al. (2023), such an approach is not applicable to fact-checking machine-generate news stories and novel human-in-the-loop methods may need to be developed to check such claims.

The rest of the paper is organized as follows. In Section 2, we discuss related work. In Section 3, we present our method for generating news stories, extract “atomic” claims, using LLMs to assess the veracity of these stories and claims, and manually verifying the assessments performed by LLMs themselves. We present experimental results in Section 4, and discuss these results in Section 5. We conclude in Section 6.

Our experiment on fact-checking LLM-generated news stories consists of four steps. First, we use two LLMs to generate a set of news stories with varying levels of factual inaccuracy. Next, from these stories, we manually extract verifiable atomic claims and decontextualize them, creating standalone claims that can be verified independently of the original story. In the third step, we prompt each LLM to evaluate the veracity of news stories generated by itself or the other LLM, as well as to assess the individual atomic claims. Finally, we conduct a human evaluation to determine the accuracy of the LLMs’ veracity assessments.

We conduct a comprehensive set of experiments to evaluate the performance of GPT-4o and GLM models in verifying claims within generated news articles. Both models are assessed in the contexts of local and national news generation, with claim verification performed across all relevant dimensions. For the claim verification task, we classify the assessment results into five possible categories, as outlined below:

• •

  Correct Assessment (CA): The model correctly identifies the veracity of the claim without providing a rationale.

• •

  Correct Assessment and Correct Reasoning (CA/CR): The model correctly identifies the veracity of the claim and provides a correct justification for its assessment.

• •

  Correct Assessment and Wrong Reasoning (CA/WR): The model correctly classifies the claim but with flawed reasoning.

• •

  Wrong Assessment (WA): The model incorrectly classifies the veracity of the claim.

• •

  No Assessment (NA): The model fails to provide any assessment.

For examples of each type of assessment, please see Appendix A.4.

In our evaluation of LLMs’ ability to assess the veracity of LLM-generated news articles and claims, we find that LLMs perform better when evaluating claims in national news compared to local news. They are also more accurate at assessing factually correct claims than factually wrong ones. Additionally, LLMs excel at evaluating claims expressed as linguistic states rather than those describing dynamic events. These seemingly distinct observations can be traced back to a common underlying factor: LLMs are more effective at processing well-documented, high-frequency information that is more likely to have been included in their training data. National news claims are typically better documented than local news claims, linguistic states are more stable and frequently recorded than rapidly evolving dynamic events, and factually accurate claims are more likely to appear in the training data than factually false ones.

Using RAG significantly increases the level of correct assessments, but it also leads to a higher number of wrong assessments due to irrelevant search results (55 out of 92 cases), no search results (10 out of 92 cases), or wrong reasoning (27 out of 92 cases). Examples of each of these cases can be found in Appendix A.5. There are still a significant number of no assessments (NA) even with the RAG approach, either because no search results are retrieved or because the search results are noisy and irrelevant.

Focusing in on cases where the assessment is correct but the reasoning is wrong (CAWR cases), we find that 4 out of 11 cases stem from irrelevant search results, 6 out of 11 cases are due to missing search results, where the model still arrived at the correct assessment but without proper justification. Only 1 case results from a pure reasoning failure, which means that the model misapplied its logic despite having relevant evidence. This suggests that errors in verification are largely due to weak or absent supporting evidence rather than purely logical failures within the model.

Our analysis of CAWR cases further reveals that when the model lacks access to reliable supporting evidence, it tends to provide speculative or inconsistent reasoning. Specifically, when faced with unverifiable claims, the model struggles to construct sound reasoning, often defaulting to generic or misleading explanations. In addition, when retrieval returns misleading or tangentially related documents, the model may incorporate incorrect details into its justification, amplifying reasoning errors. This underscores the need for future research on fact-checking machine-generated news content to prioritize the retrieval of precise and reliable evidence.

RAG systems also have the tendency to venture guesses even in the absence of evidential support, and this is problematic even if the guess is correct. For claims the retrieval system cannot find evidence for, human-in-the-loop approaches may need to be developed to ensure accuracy and reliability.

Our study uses claims that are manually extracted and decontextualized. Fully automatic evaluation systems would require that the atomic claims are automatically extracted and decontextualized, with the goal of extracting all and only checkable claims from an LLM-generated text. This is especially challenging for news stories, which may contain vague and subjective language. Unlike structured biographical datasets like FactScore Min et al. (2023), where factual claims are easy to verify against Wikipedia, news stories contain vague and context-dependent details that require more sophisticated reasoning. For automatic fact-checking systems to gain the trust and confidence of users, it is critical for such reasoning process to be transparent and interpretable.

We conducted a diagnostic study to evaluate the strengths and limitations of using LLMs and RAG systems for fact-checking claims in machine-generated "news" reports. While these systems can verify the veracity of a significant portion of claims (nearly 70%), a considerable number are either incorrectly assessed or left unassessed due to irrelevant retrievals, flawed reasoning, or insufficient evidence. This issue is particularly pronounced for rare claims with limited evidential support, which are common in news reports. Our findings underscore the need for more precise and reliable retrieval systems and the incorporation of human-in-the-loop approaches when evidence is unavailable. Future work will explore the ability of LLMs to generate verifiable claims, a crucial step toward fully automated fact-checking systems.

In this diagnostic study, we relied on manually extracted claims, which inherently limits the size of the dataset and, consequently, the breadth of the analysis. While our dataset comprises 92 news articles and 1,337 individual claims, covering a diverse range of factual errors, we acknowledge that its size is a limitation. The manual extraction process is time-consuming and labor-intensive, making it challenging to scale the dataset to include a larger number of claims. Despite this limitation, we carefully curated the dataset to ensure it is representative of the types of claims commonly found in machine-generated news reports. As a result, we are confident that the dataset is sufficiently large and diverse to support reliable and meaningful conclusions. Data will be made available on request.

Machine-generated news reports can pose significant risks if they are mistaken for authentic, factual content. To mitigate these risks, when releasing the dataset for our study, we will ensure that it is clearly labeled as machine-generated and explicitly highlight that it contains false claims. This labeling is critical to prevent misuse of the dataset and to maintain transparency for researchers, developers, and the broader community. By doing so, we aim to promote ethical research practices and minimize any potential harm arising from the dissemination of this data.

In the NLP community, it is common practice to release datasets publicly by hosting them on open-source platforms like GitHub. However, in this case, it is more appropriate to store the data on a private server and provide access to fellow researchers upon request. This approach is preferable for two key reasons. First, releasing the data on an open-source platform risks it being incorporated into the training data of future LLM versions, rendering results non-comparable. Second, the dataset is primarily useful to researchers and serves little to no practical purpose for the general public.