Instruct Large Language Models to Generate Scientific Literature Survey Step by Step

In this paper, we address the NLPCC 2024 Scientific Literature Survey Generation evaluation task, which focuses on generating literature surveys based on given subjects and reference papers.

Specifically, illustrated in Fig. 1, given a list of reference papers $R=\{r_{1},r_{2},...,r_{n}\}$ and a list of key subjects (topics) $S=\{s_{1},s_{2},...,s_{m}\}$, our goal is to produce a well-structured literature survey $\left<T,A,M\right>$. The output includes the title $T$, abstract $A$, and the main body $M=\left\{(h_{1},c_{1}),(h_{2},c_{2}),...,(h_{k},c_{k})\right\}$, where $h_{i}$ represents the headings at different levels and $c_{i}$ denotes the corresponding contents.

Notice that we did not utilize the reference content field provided in the test set because the NLPCC 2024 task guideline claimed that only the subject and reference were provided for testing. Additionally, we did not employ any retrieval-based methods to augment the reference content based on the titles of the reference papers, as the guidelines explicitly state that data other than the training set cannot be used in the process of model development.

The framework for our step-by-step literature survey generation is illustrated in Fig. 2. The generation pipeline consists of six steps, divided into two main phases.

In the first phase, outline generation, we instruct the LLM to sequentially generate the title, section headings, and abstract. Each step is conditioned on all previous outputs to ensure a coherent planning of the entire literature survey.

In the second phase, subsection and content generation, we select references for each section and then generate subsection headings and content based on the generated outline and the selected references. This design ensures that the generation of each subsection is aware of the overall structure of the paper, while also reducing the input length to save costs on LLM API usage.

In this section, we will elaborate on the prompt designs for each of the six steps in Fig. 2.

We use the dataset from the NLPCC 2024 Scientific Literature Survey Generation evaluation task, provided by Kexin Technology [9]. The dataset comprises randomly crawled arXiv survey papers along with their references, containing 500 survey papers in the training set and 200 in the test set. Since the evaluation task did not designate specific instances for validation rather than training, we have not included a separate validation set in our statistics.

In the training set, the fields including title, article id, subject, abstract, content, reference, and reference content are provided. While in the test set, only subject, reference, and reference content are provided. The statistics of the dataset is shown in Table. 1.

In this paper, we only use the training set for prompt design, instead of using them for parameter tuning or in-context learning. As discussed in §3.1, because of the requirements in task guideline, we did not utilize the reference content field provided in the test set, nor did we employ any retrieval-based methods to augment the reference titles.

We implemented our system using the Qwen-long version of Alibaba’s Tongyi Qianwen model [5], with a cost of 0.5 RMB per million tokens for input and 2 RMB per million tokens for output. For instances where the reference list is too long for Qwen-long to process (with the role of user), we truncate the references, leaving only the first 80-100 references. Additionally, in cases where the reference list contains inappropriate content that Qwen-long refuses to process, we try to use references with odd or even indices to circumvent this issue. Generating all 200 surveys in the test set cost 20.86 RMB, approximately 0.10 RMB per paper. This low cost enhances the practicality of our method.

To evaluate the quality of the generated literature survey, we use three kinds of metrics:

ROUGE [18] is a recall-oriented reference-based metrics. Specifically, we employ ROUGE1, ROUGE2, and ROUGEL scores. We use the implement provided by Google¹¹1https://github.com/google-research/google-research/tree/master/rouge.

Soft Heading Recall (S-H Recall for short) is used to evaluate the structure of the generated survey.

$T=\{t_{1},t_{2},t_{3},\ldots,t_{K}\}$ represents a group of the chapter titles/heads in a generated/reference survey. $R$ and $G$ are the chapter titles of the generated and reference survey, respectively. The bge-large-en-v1.5²²2https://huggingface.co/BAAI/bge-large-en-v1.5 model is used for text embedding. This score encourages the similarity between generated and reference chapter titles while punishes the similarity of titles within the generated survey.

Human evaluation is also used to manually evaluate the following aspects:

• •

  Fluent language with clear expression;

• •

  Logical article structure;

• •

  Ample, reliable, and accurate citations;

• •

  Consistency of content with the theme, staying on-topic;

• •

  Broad analytical scope.

The overall score is calculated as the average of the ROUGE score, the soft heading recall, and the human score. Note that the ROUGE score here is the average of ROUGE1, ROUGE2, and ROUGEL.

The official evaluation results is shown in Table 2. We can see that our system, ID, achieved the third place in the evaluation task, with an overall score only 0.03% lower than the second-place team.

Additionally, our system demonstrates competitive performance across other metrics. Notably, in soft heading recall, our system achieves a score of 95.84%, securing second place and trailing the leading team by just 1.17%. This result highlights its effectiveness in generating high-quality headings. We attribute this success to our sequential generation design, which includes an individual heading generation step, providing a high-level perspective that enhances the ability of large language models to generate meaningful headings.

However, human evaluation remains a relatively weak aspect of our method. Without leveraging the content of the reference papers, the hallucination phenomenon of LLMs becomes significant. Consequently, our method cannot ensure the accuracy and reliability of the citations and analysis in the generated survey, thereby impairing our human evaluation score. In the future, we will incorporate the reference content into our framework to improve the factual accuracy and reliability of our outputs, aiming to generate more accurate and trustworthy literature surveys.

An example of our generated survey paper and the corresponding reference one is shown in Table 3. Our system successfully captures the subject of the paper, interpretability in neural networks, and generates a relevant title.

For the outline generation, this example achieves a soft heading recall of 97.78%. While the outline produced by our system is somewhat lengthy and too detailed, it successfully captures key aspects of the survey, such as explainable AI, Saliency Maps, and Black-Box Models. The discussion on ethical considerations and future directions at the end of the survey is also well-founded. In the future, some specific instruction strategies can be designed to encourage the LLM to generate more concise outlines.

For the abstract and content (citation), we can see that our generated paper captures several common points with the reference one (bold in Table. 3), resulting in appropriate abstracts and citations. However, some citations are still influenced by hallucinations, which could be mitigated by incorporating the detailed content of the reference papers.