CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Support

We leverage the Cochrane Database of Systematic Reviews²²2https://www.cochranelibrary.com/cdsr/reviews to obtain sets of related studies, since the systematic review writing process requires experts to extensively search for and curate studies relevant to review topics. We obtain all systematic reviews and the corresponding studies included in each review from the Cochrane website Wallace et al. (2020). We then filter this set of systematic reviews to only retain those including at least 15 and no more than 50 corresponding studies. We discard reviews with very few studies since a hierarchical organization is unlikely to provide much utility, while reviews with more than 50 studies are discarded due to the inability of LLMs to effectively handle such long inputs Liu et al. (2023). Our filtering criteria leave us with 472 systematic reviews (or sets), each including an average of 24.7 studies, which serve as input to our hierarchy generation pipeline.

Prior work on using LLMs for complex tasks has shown that decomposing the task into a series of steps or sub-tasks often elicits more accurate responses Kojima et al. (2022); Wei et al. (2022b); D’Arcy et al. (2024). Motivated by this, we decompose hierarchy generation from a set of related scientific studies into three sub-tasks: (i) compressing study findings into concise claims, (ii) initiating hierarchy generation by generating root categories, and (iii) completing hierarchy generation by producing remaining categories and organizing claims under them. Our hierarchy generation pipeline consists of a pre-generation module that tackles task (i) and a hierarchy proposal module that handles tasks (ii) and (iii) (see Figure 2)). Additionally, our pipeline can generate multiple (up to five) potential hierarchies per topic. We describe our pipeline module in further detail below and provide complete prompt details in Appendix A.

In this sub-task, given all parent-child category links from a hierarchy (e.g., $P1\rightarrow S[1-5]$ in Figure 1), for each link, humans are prompted to determine whether the child is a valid sub-category of the parent. Annotators can label parent-child category links using one of the following labels: (i) parent and child categories have a hypernym-hyponym relationship (e.g., exercise modalities $\rightarrow$ aerobic exercise), (ii) parent and child categories are not related by hypernymy but the child category provides a useful breakdown of the parent(e.g., aerobic exercise $\rightarrow$ positive effects), and (iii) parent and child categories are unrelated (e.g., aerobic exercise $\rightarrow$ anaerobic exercise). Categories (i) and (ii) are positive labels indicating valid links, while category (iii) is a negative label capturing incorrect links in the existing hierarchy.

For a hierarchical organization to be useful, in addition to validity of parent-child category links, all sibling categories (i.e., categories under the same parent, like $S1-S5$ in Figure 1) should also be coherent. Therefore, in our second sub-task, given a parent and all its child categories, we ask annotators to determine whether these child categories form a coherent sibling group. Annotators can assign a positive or negative coherence label to each child category in the group. For example, given the parent category “type of cancer” and the set of child categories “liver cancer”, “prostate cancer”, “lung cancer”, and “recurrence”, the first three categories are assigned positive labels, while “recurrence” is assigned a negative label since it is not a type of cancer. All categories assigned a negative label capture incorrect groups in the existing hierarchy.

Unlike the previous sub-tasks which focus on assessing links between categories at all levels of the hierarchy, the final sub-task focuses on assessing the assignment of claims to various categories. Given a claim and all categories present in the hierarchy, for each claim-category pair, humans are prompted to assess whether the claim contains any information relevant to that category. The claim-category pair is assigned a positive label if relevant information is present, and negative otherwise. For every category, we include the path from the root to provide additional context which might be needed to interpret it accurately (e.g., “positive findings” has a broader interpretation than “chemotherapy $\rightarrow$ positive findings”). Instead of only assessing relevance of categories under which a claim has currently been categorized, this sub-task evaluates all claim-category pairs in order to catch recall errors, i.e., cases in which a claim could be assigned to an category but is not categorized there currently.

An additional benefit of collecting corrections for preliminary hierarchies (as described above) is that this data allows us to quantify the quality of our LLM-generated hierarchies and measure the performance of our hierarchy generation pipeline.

These characteristics indicate that our LLM-generated hierarchies have interesting structural properties.

We briefly discuss the experimental setup we use to evaluate whether model performance on sibling coherence and claim categorization correction can be improved using our collected feedback data.

Table 3 presents the performance of all models on the task of identifying sibling groups that are incoherent. Finetuning models on this task is challenging due to the small size of the training set ($n=298$) and imbalanced labels. Despite upsampling and model selection based on precision, finetuned Flan-T5 models do not perform well on this task (best F1-score of $33.3\%$). Additionally LLMs also do not perform well despite the use of chain-of-thought prompting to handle the complex reasoning required for this task. At $51.5\%$ F1, LLMs outperform finetuned models; however, their precision ($46.7\%$ for GPT-4-Turbo) is still not good enough to detect incoherent sibling groups confidently. These results indicate that this correction sub-task is extremely difficult to automate and will likely continue to require expert intervention.

Table 4 shows the performance of all models on the task of correcting assignment of claims to categories in the hierarchy. Following the strategy described in §3.5, given a claim, we first use our models to generate predictions for every claim-category pair (all category nodes) and then obtain the final label for each category by applying an AND operation over all predictions from the root category to that category. Our results show that this task is easier to automate—fine-tuning Flan-T5 on our collected training dataset leads to better scores on all metrics compared to our LLM pipeline. Crucially, recall which is much more time-consuming for humans to fix, improves by $15.9$ points using Flan-T5-large indicating that automating this step can provide additional efficiency gains during correction. LLMs perform well too, with GPT-4-Turbo achieving the best recall rate among all models, but its lower precision score makes the predictions less reliable overall.

Interestingly, we notice that all models perform better on the OOD test for both correction tasks, indicating that the OOD test set likely contains instances that are less challenging than the ID set.

Comparing the claim categorization predictions of Flan-T5-large on our test set with our LLM-based hierarchy generation pipeline reveals that it flips labels in $24.7\%$ cases, of which 63.5% changes are correct. This indicates that a FLAN-T5-large corrector can potentially improve claim categorization of LLM-generated hierarchies. Therefore, we apply this corrector to the remaining 372 LLM-generated hierarchies that we do not have expert corrections for to improve claim assignment for those. Our final curated dataset CHIME contains hierarchies for 472 research topics, of which hierarchies for 100 topics have been corrected by experts on both category linking and claim categorization, while hierarchies for the remaining 372 have had claim assignments corrected automatically.

Prior work on developing literature review support tools has largely focused on using summarization techniques for end-to-end review generation or to tackle specific aspects of the problem (see Altmami and Menai (2022) for a detailed survey). Some studies have focused on generating “citation sentences” discussing relationships between related papers, which can be included in a literature review Xing et al. (2020); Luu et al. (2021); Ge et al. (2021); Wu et al. (2021). Other work has focused on the task of generating related work sections for a scientific paper Hoang and Kan (2010); Hu and Wan (2014); Li et al. (2022); Wang et al. (2022), which while similar in nature to literature review, has a narrower scope and expects more concise generation outputs. Finally, motivated by the ever-improving capabilities of generative models, some prior work has attempted to automate end-to-end review generation treating it as multi-document summarization, with limited success Mohammad et al. (2009); Jha et al. (2015); Wallace et al. (2020); DeYoung et al. (2021); Liu et al. (2022); Zhu et al. (2023). Of these, Zhu et al. (2023) generates intermediate hierarchical outlines to scaffold literature review generation, but unlike our work, they do not produce multiple organizations for the same set of related studies. Additionally, we focus solely on the problem of organizing related studies for literature review, leaving review generation and writing assistance to future work.

Organizing document collections is an extensively-studied problem in NLP, with several classes of approaches such as clustering and topic modeling Dumais et al. (1988) addressing this goal. Despite their utility, conventional clustering and topic modeling approaches are not easily interpretable Chang et al. (2009), requiring manual effort which introduces subjectivity and affects their reliability Baden et al. (2022). Recent work has started exploring whether using LLMs for clustering Viswanathan et al. (2023); Zhang et al. (2023); Wang et al. (2023) and topic modeling Pham et al. (2023) can alleviate some of these issues, with promising results. This motivates us to experiment with LLMs for generating hierarchical organizations of scientific studies. Interestingly, TopicGPT Pham et al. (2023) also attempts to perform hierarchical topic modeling, but is limited to producing two-level hierarchies unlike our approach which generates hierarchies of arbitrary depth.