NovAScore : A New Automated Metric for
Evaluating Document Level Novelty

provides a straightforward approach to evaluating ACU novelty. We compare each target ACU with historical ACUs from the ACUBank. If any historical ACU exceeds a set similarity threshold with the target ACU, it is classified as non-novel, indicating likely repetition; and vice versa. This method efficiently assesses overlap, making it a practical tool for novelty detection.

is based on the principle that a premise $P$ entails a hypothesis $H$ if a typical human reader would conclude that $H$ is most likely true after reading $P$ Dagan et al. (2005). In the context of novelty detection, this means that if one or more entailing premises are found for a given hypothesis, the content of that hypothesis is considered not new Bentivogli et al. (2011). To evaluate ACU-level novelty, we concatenate the most relevant historical ACUs into a single premise and compare it against the target ACU as the hypothesis. If the historical content entails the target ACU, it is classified as non-novel; otherwise, it is considered novel.

approach is widely used to evaluate information overlap by representing reference information as question-answer pairs, with recall assessed by how well the candidate text answers these questions Deutsch et al. (2021); Zhong et al. (2021). We adapt this method in reverse: for each target ACU, we generate questions where the target ACU itself is the answer. If any historical ACUs can answer these questions, the target ACU is considered non-novel; otherwise, it is novel. We generate three questions per ACU, focusing on named entities and noun phrases Deutsch et al. (2021). The answers derived from historical ACUs are consolidated into a single sentence and compared to the target ACU. If the consolidated answer has a cosine similarity of 0.85 or higher with the target ACU, it is classified as non-novel. The rationale for this threshold is detailed in Appendix A.2.2.

We utilize the following two datasets, which, to the best of our knowledge, are among the few publicly available in the news domain:

• 1.

  TAP-DLND 1.0 Ghosal et al. (2018b): This dataset contains 2,736 human-annotated novel documents and 2,704 non-novel documents, all clustered into specific categories. Each novel or non-novel document is annotated against three source documents.

• 2.

  APWSJ Zhang et al. (2002): This dataset comprises 10,833 news articles from the Associated Press (AP) and Wall Street Journal (WSJ) corpora, covering 33 topics. The documents are chronologically ordered and annotated into three categories: absolutely redundant, somewhat redundant, and novel. Of these, 7,547 are novel, 2,267 are somewhat redundant, and 1,019 are absolutely redundant.

For both datasets, we sample 500 documents. In TAP-DLND 1.0, we randomly select 500 documents across clusters. For APWSJ, where documents are chronologically sorted and annotated for novelty relative to earlier ones, we sequentially select a balanced set of 500 non-novel and novel documents. Table 1 provides the dataset statistics used in our experiments.

We utilize GPT-4o across all modules, including ACU extraction, document summarization, salient ACU selection, and both NLI and QA-based novelty evaluation. Details on the prompts are provided in Appendix A.1.

The ACUBank is implemented using FAISS²²2Link to FAISS. for fast similarity search and efficient indexing. Each ACU is indexed by its sentence embedding from the pre-trained SentenceBERT³³3Model card of all-mpnet-base-v2.. In TAP-DLND 1.0, we create separate databases for each of the 223 clusters. For APWSJ, documents are processed chronologically into a unified database.

To retrieve relevant historical ACUs from the ACUBank, we select the top-5 ACUs with a cosine similarity of 0.6 or higher. The rationale for this threshold is detailed in Appendix A.2.2. If any meet this threshold, the ACU is considered non-novel when using the cosine similarity novelty evaluator. For NLI or QA novelty evaluators, these similar ACUs are concatenated to form the premise for NLI or the context for QA, further assessing the ACU’s novelty.

We use different hyperparameters for each dataset in the dynamic weight adjustment to account for ACU salience when calculating the overall NovAScore. These parameters control the devaluation of non-salient ACUs and adjust for extreme salience ratios, varying by dataset. For TAP-DLND 1.0, we use $\alpha=0$, $\beta=0.5$, and $\gamma=1$ (no adjustment). For APWSJ, we use $\alpha=1$, $\beta=0.5$, and $\gamma=0.7$. The rationale for these choices is detailed in Appendix A.2.1.

We employ Point-Biserial, Spearman, and Kendall to evaluate the relationship between the NovAScore and human annotations of document-level novelty. Point-Biserial is a special case of Pearson correlation that compares continuous variables with binary variables. For the TAP-DLND 1.0 dataset, where human annotations are binary, we assign a label of 1 to novel and 0 to non-novel for calculating correlations. In contrast, the APWSJ dataset contains three classes: we assign novel a value of 1, somewhat redundant a value of 0.5, and absolute redundant a value of 0 for the Spearman and Kendall correlations. For Point-Biserial correlation on APWSJ, we set both absolute redundant and somewhat redundant to 0.

As shown in Table 2, across both datasets and using all novelty evaluators, NovAScore demonstrates a moderate to strong correlation with human judgments of document-level novelty, as indicated by the correlation cutoffs detailed in Appendix A.3. These correlations are statistically significant, with $p$-values ranging from $10^{-22}$ to $10^{-56}$, indicating the robustness of NovAScore in aligning with human perceptions of novelty.

Among the different evaluators, the NLI novelty evaluator consistently outperforms the others, showing a particularly strong correlation with human annotations. Notably, on the TAP-DLND 1.0 dataset, the NovAScore with the NLI novelty evaluator achieves a Point-Biserial of 0.626, a Spearman of 0.622, and a Kendall of 0.510, all signifying a strong alignment with human judgment.

Since existing public datasets only provide single categorical labels at the document level without fine-grained annotations, we curate and annotate a new dataset for this purpose. We manually select 32 news articles, clustered into 8 topics. Within each topic, the documents are sorted in chronological order, and we extract ACUs using GPT-4o, following the strategy described in Sections 3.1. Human annotators evaluate each ACU based on four labels: correctness (logical and factual consistency), redundancy (non-informativeness or intra-document non-novelty), novelty, and salience. The full annotation instructions and label schema are provided in Appendix B.1. Two annotators independently perform the entire annotation task. After completing the annotations, the annotators meet to discuss and resolve any conflicting annotations, ensuring consensus on the final labels. Further discussion on annotation quality is presented in Appendix B.2.

We compare the performance of each novelty evaluator against human annotations of ACU-level novelty. As shown in Table 3, all novelty evaluators achieve strong classification results, with the NLI-based evaluator leading with an accuracy of 0.94.

We aggregate ACU-level scores to compute the document-level novelty score, resulting in the following NovAScore variants: (1) NovAScore${}_{\text{human}}$, using human-annotated novelty and salience labels, and (2) NovAScore${}_{\text{CosSim}}$, NovAScore${}_{\text{NLI}}$, and NovAScore${}_{\text{QA}}$, which are fully automated versions utilizing their respective novelty evaluators and GPT salience evaluator. For all variants, we apply weight adjustment parameters of $\alpha=1$, $\beta=0.5$, and $\gamma=0.7$, as used for the APWSJ dataset. We also compute all variants without incorporating salience to better assess the performance of the novelty evaluators.

We then use Pearson correlation to evaluate the relationship between NovAScore${}_{\text{human}}$ and the three automated variants, as all produce continuous scores with nearly linear distributions. As shown in Table 4, all automated variants demonstrate strong to very strong correlations, with NovAScore${}_{\text{NLI}}$ achieving the highest Pearson correlation of 0.920 without salience and 0.835 with salience. The discrepancy between human-annotated and GPT-selected salient ACUs, as discussed in Section 5.1, slightly degrades the correlation statistics when salience is incorporated, which is expected. Despite this, the fully automated NovAScore with salience included still shows a very strong correlation with human judgments, indicating that GPT-4o is a reasonable estimator of information salience. The full correlation results between the three automated variants and NovAScore${}_{\text{human}}$ are detailed in Table 8 in Appendix A.4.

These results underscore the effectiveness of NovAScore in capturing document-level novelty while also providing fine-grained interpretability at the atomic level, making it a reliable tool for assessing document-level novelty.

As introduced in Section 4.2, we collect correctness and redundancy labeled on GPT-generated ACUs during human annotation. Results reveal that none of the GPT-4o generated ACUs are labeled as incorrect by the annotators, and only 0.1% are considered redundant. These findings indicate that GPT-4o is highly reliable in generating high-quality ACUs.

Currently, no public datasets are designed for abstractive document-level ACU extraction. The closest are summarization datasets, which focus on key information and miss non-salient ACUs. Thus, our evaluation prioritizes precision over recall, leaving full ACU extraction and novelty recall for future work (see Limitations section).

Recent studies suggest that LLM-generated summaries are often on par with human-written ones Zhang et al. (2024), supporting our confidence in GPT-4o’s ability to evaluate salience. We compare GPT-selected salient ACUs with human-annotated salience labels as outlined in Section 4.2, and find that GPT achieves a macro F1 score of 0.6. This discrepancy may result from the different conditions: human annotators determine salience in real-time without summaries, making the task more challenging, while GPT-4o can reference the generated summary. Additionally, salience is inherently subjective, making it difficult to standardize. Despite these factors, GPT-4o’s performance in salience evaluation is satisfactory.