Accelerating Clinical Evidence Synthesis with Large Language Models

A systematic understanding of cancer treatments is crucial for oncology drug discovery and development. We retrieved a list of cancer treatments from the National Cancer Institute’s introductory page as the keywords to search medical systematic reviews [22]. To ensure data quality, we crafted comprehensive queries with automatic filtering and manual screening. For each review, we obtained the list of studies with their PubMed IDs, retrieved their full content, and extracted study characteristics and clinical outcomes. We followed PubMed’s usage policy and guidelines during retrieval. Further manual checks were performed to correct inaccuracies, eliminate invalid and duplicate papers, and refine the text for clarity (Methods). The final TrialReviewBench dataset consists of 2,220 studies involved in 100 reviews (Fig. 2a), covering four major topics: Immunotherapy, Radiation/Chemotherapy, Hormone Therapy, and Hyperthermia. We manually created three major evaluation tasks based on these reviews: study search, study screening, and data extraction.

The study search task begins with the PICO (Population, Intervention, Comparison, Outcome) elements extracted from the abstract of a systematic review, which serve as the formal definition of the research question. The model being tested is tasked with generating relevant keywords for the treatment and condition terms, as depicted in Fig. 2e. These keywords are then used to form Boolean queries, which are submitted to search citations in the PubMed database. The performance of the model is evaluated by checking whether the retrieved studies include those that were actually involved in the target systematic review. The recall rate is computed by measuring the proportion of actually involved studies identified through the search.

For the study screening task, the input consists of the PICO elements defined in the target systematic review. A candidate set of 2,000 citations is created by combining the actual studies included in the review with additional citations retrieved during the search but not included in the review. The model being tested ranks these citations based on the likelihood that each citation should be included in the systematic review. To assess the model’s performance, we compute Recall@$k$: the recall value indicating how many of the actual included studies appear in the top $k$ ranked candidates.

The data extraction task focuses on retrieving specific information from the input study documents. In this case, we extract Table 1 from each systematic review, which typically details study characteristics such as study design, population demographics, and outcome measurements. These characteristics are matched to the individual studies and manually verified, yielding 1,334 study characteristic annotations. Additionally, we extract individual study results from the review’s reported analysis, often presented in forest plots, capturing metrics such as overall response and event rates, resulting in 1,049 study result annotations. The model being tested is given a list of target data points to extract, and its output is evaluated by assessing the accuracy of the extracted information based on the annotated datasets.

Large language models (LLMs) excel in adapting to new tasks when provided with task-specific prompts while often struggling with complex tasks that require multiple steps of planning and reasoning. Additionally, interacting and collaborating with LLMs can be problematic due to their opaque nature and the complexity of debugging [23]. In this study, we developed TrialMind that decomposes the clinical evidence synthesis process into four main tasks (Fig. 1 and Methods). Initially, using the provided research question enriched with population, intervention, comparison, and outcome (PICO) elements, TrialMind conducts a comprehensive search from the literature. It also works with users to build the eligibility criteria for target studies and then automate screening and ranking identified citations. Next, TrialMind browses the study details to extract the study characteristics and pertinent findings. To ensure the accuracy and integrity of the data, each output is linked to the sources for manual inspection. In the final step, TrialMind standardizes the clinical outcomes for meta-analysis.

Finding relevant studies from medical literature like PubMed, which contains over 35 million entries, can be challenging. Typically, this requires the research expertise to craft complex queries that comprehensively cover pertinent studies. The challenge lies in balancing the specificity of queries: too stringent, and the search may miss relevant studies; too broad, and it becomes impractical to manually screen the overwhelming number of results. Previous approaches propose to prompt LLMs to generate the searching query directly [9], which can induce incomplete searching results due to the limited knowledge of LLMs. In contrast, TrialMind is designed to produce comprehensive queries through a pipeline that includes query generation, augmentation, and refinement. It also provides users with the ability to make further adjustments (Fig. 2b).

The dataset involving clinical studies spanning ten cancer treatment areas was used for evaluation (Fig. 2a). For each review, we collected the involved studies’ PubMed IDs as the ground-truth and measured the Recall, i.e., how many ground-truth studies are found in the search results. We created two baselines as the comparison: GPT-4 and Human. The GPT-4 baseline makes a guided prompt for LLMs to generate the boolean queries [9]. It represents the common way of prompting LLMs for literature search query generation. The Human baseline represents a way where the key terms from PICO elements are extracted manually and expanded, referring to UMLS [24], to construct the search queries.

Overall, TrialMind achieved a Recall of 0.782 on average for all reviews in TrialReviewBench, meaning it can capture most of the target studies. By contrast, the GPT-4 baseline yielded Recall = 0.073, and the Human baseline yielded Recall = 0.187. We divided the search results across four topics determined by the treatments studied in each review (Fig. 2c). Our analysis showed that TrialMind can identify many more studies than the baselines. For instance, TrialMind achieved $\text{Recall}=0.797$ with identified studies $N$ = 22,084 for Immunotherapy-related reviews, while the GPT-4 baseline got $\text{Recall}=0.094$ ($N$ studies = 27), and the Human baseline got $\text{Recall}=0.154$ ($N$ studies = 958), respectively. In Radiation/Chemotherapy, TrialMind achieved $\text{Recall}=0.780$, the GPT-4 baseline got $\text{Recall}=0.020$, and the Human baseline got $\text{Recall}=0.138$. In Hormone Therapy, TrialMind achieved $\text{Recall}=0.711$, the GPT-4 baseline got $\text{Recall}=0.067$, and the Human baseline got $\text{Recall}=0.232$. In Hyperthermia, TrialMind achieved $\text{Recall}=0.834$, the GPT-4 baseline got $\text{Recall}=0.106$, and the Human baseline got $\text{Recall}=0.202$. These results demonstrate that regardless of the search task’s complexity, as indicated by the variability in the Human baseline, TrialMind consistently retrieves nearly all target studies from the PubMed database. This robust performance provides a solid foundation for accurately identifying target studies in the screening phase.

Furthermore, we made scatter plots of Recall versus the number of target studies for each review (Fig. 2d). The hypothesis was that an increase in target studies correlates with the difficulty of achieving complete coverage. Our findings reveal that TrialMind consistently maintained a high Recall, significantly outperforming the best baselines across all 100 reviews. A trend of declining Recall with an increasing number of target studies was confirmed through regression analysis. It was found that the GPT-4 baseline struggled, showing Recall close to 0, and the Human baseline results varied, with most reviews below 0.5. As the number of target studies increased, the Human and GPT-4 baselines’ Recall decreased to nearly zero. In contrast, TrialMind demonstrated remarkable resilience, showing minimal variation in performance despite the increasing number of target studies. For instance, in a review involving 141 studies, TrialMind achieved a Recall of 0.99, while the GPT-4 and Human baselines obtained a Recall of 0.02 and 0, respectively.

Typically, human experts manually sift through thousands of retrieved studies to select relevant ones for inclusion in a systematic review. This process adheres to the PRISMA statement [21], which involves creating a list of eligibility criteria and assessing each study’s eligibility. TrialMind streamlines this task through a three-step approach: (1) it generates a set of inclusion criteria, which are subject to user’s adjustments; (2) it applies these criteria to evaluate the study’s eligibility, denoted by $\{-1,0,1\}$ where $-1$ and $1$ represent eligible and non-eligible, and $0$ represents unknown/uncertain, respectively; and (3) it ranks the studies by aggregating the eligibility predictions, where the aggregation strategy can be specified by users (Fig. 3a). We took a summation of the criteria-level eligibility predictions as the study-level relevance prediction scores for ranking. As such, TrialMind provides a rationale for the relevance scores by detailing the eligibility predictions for each criterion.

We chose MPNet [25] and MedCPT [26] as the general domain and medical domain ranking baselines, respectively. These methods compute study relevance by the cosine similarity between the encoded PICO elements as the query and the encoded study’s abstracts. We also set a Random baseline that randomly samples from candidates. We created the evaluation data based on the search results in the first stage. For each review, we mixed the target studies with the other found studies to build a candidate set of 2,000 studies for ranking. Discriminating the target studies from the other candidates is challenging since all candidates meet the search queries, meaning they most probably investigate the relevant therapies or conditions. We evaluated the ranking performance using the Recall@20 and Recall@50 metrics. The concatenation of the title and abstract of each study is used for all methods as inputs.

We found that TrialMind greatly improved ranking performances, with the fold changes over the best baselines ranging from 1.3 to 2.6 across four topics (Table 3c). For instance, for the Hormone Therapy topic, TrialMind obtained $\text{Recall}@20=0.431$ and $\text{Recall}@50=0.674$. In the Hyperthermia topic, TrialMind obtained $\text{Recall}@20=0.518$ and $\text{Recall}@50=0.710$. In the Immunotherapy topic, TrialMind obtained $\text{Recall}@20=0.567$ and $\text{Recall}@50=0.713$. In the Radiation/Chemotherapy topic, TrialMind obtained $\text{Recall}@20=0.416$ and $\text{Recall}@50=0.654$. In contrast, other baselines exhibit significant variability across different topics. The general domain baseline MPNet was the worst as it performed similarly to the Random baseline in $\text{Recall}@20$. MedCPT showed marginal improvement over MPNet in the last three topics, while both failed to capture enough target studies in all topics.

Furthermore, TrialMind demonstrated significant improvements over the baselines across various therapeutic areas (Fig. 3b). For example, in “Cancer Vaccines” and “Hormone Therapy,” TrialMind substantially increased $\text{Recall}@50$, achieving 33.33-fold and 10.53-fold improvements, respectively, compared to the best-performing baseline. TrialMind generally attained a fold change greater than 2 (ranging from 1.57 to 33.33). Despite the challenge of selecting from a large pool of candidates ($n=2,000$) where candidates were very similar, TrialMind identified an average of 43% of target studies within the top 50. We compared TrialMind to MedCPT and MPNet for $\text{Recall}@K$ ($K$ in 10 to 200) to gain insight into how $K$ influences the performances (Fig. 3e). We found TrialMind can capture most of the target studies (over 80%) when $K=100$.

To thoroughly assess the quality of these criteria and their impact on ranking performance, we conducted a leave-one-out analysis to calculate $\Delta\text{Recall}@200$ for each criterion (Fig. 3d). The $\Delta\text{Recall}@200$ metric measures the difference in ranking performance with and without a specific criterion, with a larger value indicating superior criterion quality. Our findings revealed that most criteria positively influenced ranking performances, as the negative influence criteria are $n=1$ in Hormone Therapy, $n=1$ in Hyperthermia, $n=5$ in Radiation/Chemotherapy, and $n=7$ in Immunotherapy. Additionally, we identified redundancies among the generated criteria, as those with $\Delta\text{Recall}@200=0$ were the most frequently observed. This redundancy likely stems from some criteria covering similar eligibility aspects, thus not impacting performance when one is omitted.

TrialMind leverages LLMs to streamline extracting study characteristics such as target therapies, study arm design, and participants’ baseline information from involved studies. Specifically, TrialMind refers to the field names and the descriptions from users and use the full content of the study documents in PDF or XML formats as inputs (Fig. 4a). When the free full content is unavailable, TrialMind accepts the user-uploaded content as the input. We developed an evaluation dataset by converting the study characteristic tables from each review paper into data points. Our dataset comprises 1,334 target data points, including 696 on study design, 353 on population features, and 285 on results. We assessed the data extraction performance using the Accuracy metric.

TrialMind demonstrated strong extraction performance across various topics (Fig. 4b): it achieved an accuracy of $\text{ACC}=0.78$ (95% confidence interval (CI) = 0.75–0.81) in the Immunotherapy topic, $\text{ACC}=0.77$ (95% CI = 0.72-0.82) in the Radiation/Chemotherapy topic, $\text{ACC}=0.72$ (95% CI = 0.63-0.80) in the Hormone Therapy topic, and $\text{ACC}=0.83$ (95% CI = 0.74-0.90) in the Hyperthermia topic. These results indicate that TrialMind can provide a solid initial data extraction, which human experts can refine. Importantly, each output can be cross-checked by the linked original sources, facilitating verification and further investigation.

Diving deeper into the accuracy across different types of fields, we observed varying performance levels. It performed best in extracting study design information, followed by population details, and showed the lowest accuracy in extracting results (Fig. 4b). For example, in the Immunotherapy topic, TrialMind achieved an accuracy of $\text{ACC}=0.95$ (95% CI = 0.92-0.96) for study design, $\text{ACC}=0.74$ (95% CI = 0.67-0.80) for population data, and $\text{ACC}=0.42$ (95% CI = 0.36-0.49) for results. This variance can be attributed to the prevalence of numerical data in the fields: fields with more numerical data are typically harder to extract accurately. Study design is mostly described in textual format and is directly presented in the documents, whereas population and results often include numerical data such as the number of patients or gender ratios. Results extraction is particularly challenging, often requiring reasoning and transformation to capture values accurately. Given these complexities, it is advisable to scrutinize the extracted numerical data more carefully.

We also evaluated the robustness of TrialMind against hallucinations and missing information (Fig. 4c). We constructed a confusion matrix detailing instances of hallucinations: false positives (FP) where TrialMind generated data not present in the input document and false negatives (FN) where it failed to extract available target field information. We observed that TrialMind achieved a precision of $\text{Precision}=0.994$ for study design, $\text{Precision}=0.966$ for population, and $\text{Precision}=0.862$ for study results. Missing information was slightly more common than hallucinations, with TrialMind achieving recall rates of $\text{Recall}=0.946$ for study design, $\text{Recall}=0.889$ for population, and $\text{Recall}=0.930$ for study results. The incidence of both hallucinations and missing information was generally low. However, hallucinations were notably more frequent in study results; this often occurred because LLMs could confuse definitions of clinical outcomes, for example, mistaking ‘overall response’ for ‘complete response.’ Nevertheless, such hallucinations are typically manageable, as human experts can easily identify and correct them while reviewing the referenced material.

The challenges in extracting study results primarily stem from (1) identifying the locations that describe the desired outcomes from lengthy papers, (2) accurately extracting relevant numerical values such as patient numbers, event counts, durations, and ratios from the appropriate patient groups, and (3) performing the correct calculations to standardize these values for meta-analysis. In response to these complexities, we developed a specialized pipeline for result extraction (Fig. 4g), where users provide the interested outcome and the cohort definition. TrialMind offers a transparent extraction workflow, documenting the sources of results along with the intermediate reasoning and calculations.

We compared TrialMind against two generalist LLM baselines, GPT-4 and Sonnet, which were prompted to extract the target outcomes from the full content of the study documents. Since the baselines can only make text extractions, we manually convert them into numbers suitable for meta-analysis [27]. This made very strong baselines since they combined LLM extraction with human post-processing. We assessed the performance using the Accuracy metric.

The evaluation conducted across four topics demonstrated the superiority of TrialMind (Fig. 4d). Specifically, in Immunotherapy, TrialMind achieved an accuracy of $\text{ACC}=0.70$ (95% CI 0.62-0.77), while GPT-4 scored $\text{ACC}=0.54$ (95% CI 0.45-0.62). In Radiation/Chemotherapy, TrialMind reached $\text{ACC}=0.65$ (95% CI 0.51-0.76), compared to GPT-4’s $\text{ACC}=0.52$ (95% CI 0.39-0.65). For Hormone Therapy, TrialMind achieved $\text{ACC}=0.80$ (95% CI 0.58-0.92), outperforming GPT-4, which scored $\text{ACC}=0.50$ (95% CI 0.30-0.70). In Hyperthermia, TrialMind obtained an accuracy of $\text{ACC}=0.84$ (95% CI 0.71-0.92), significantly higher than GPT-4’s $\text{ACC}=0.52$ (95% CI 0.39-0.65). The breakdowns of evaluation results by the most frequent types of clinical outcomes (Fig. 4e) showed TrialMind got fold changes in accuracy ranging from 1.05 to 2.83 and a median of 1.50 over the best baselines. This enhanced effectiveness is largely attributable to TrialMind’s ability to accurately identify the correct data locations and apply logical reasoning, while the baselines often produced erroneous initial extractions.

We analyzed the error cases in our result extraction experiments and identified four primary error types (Fig. 4f). The most common error was ‘Inaccurate’ extraction (n=36), followed by ‘Extraction failure’ (n=27), ‘Unavailable data’ (n=10), and ‘Hallucinations’ (n=3). ‘Inaccurate’ extractions often occurred due to multiple sections ambiguously describing the same field. For example, a clinical study might report the total number of participants receiving CAR-T therapy early in the document and later provide outcomes for a subset with non-small cell lung cancer (NSCLC). The specific results for NSCLC patients are crucial for reviews focused on this subgroup, yet the presence of general data can lead to confusion and inaccuracies in extraction. ‘Extraction failure’ and ‘Unavailable data’ both illustrate scenarios where TrialMind could not retrieve the information. The latter case particularly showcases TrialMind’s robustness against hallucinations, as it failed to extract data outside the study’s main content, such as in appendices, which were not included in the inputs. Furthermore, errors caused by hallucinations were minor. The outputs were easy to identify and correct through manual inspection since no references were provided.

We selected five systematic review studies as benchmarks and referenced the clinical evidence reported in the target studies. The baseline used GPT-4 with a simple prompting to extract the relevant text pieces that report the target outcome of interest (Methods). Manual calculations were necessary to standardize the data for meta-analysis. In contrast, TrialMind automated the extraction and standardization (Fig. 5a by (1) extracting the raw result description from the input document and (2) standardizing the results by generating a Python program to assist the calculation. The standardized results from all involved studies are then fed into the R program by human experts to make the aggregated evidence in a forest plot.

We engaged with human annotators to assess the quality of synthesized clinical evidence presented in forest plots. Each annotator was asked to evaluate the evidence quality by comparing it against the evidence reported in the target review and deciding which method, TrialMind or the baseline, produced superior results (Extended Fig. 1). Additionally, they rated the quality of the synthesized clinical evidence on a scale of 1 to 5. The assignment of our method and the baseline was randomized to ensure objectivity. The results highlighted TrialMind’s superior performance compared to the direct use of GPT-4 for clinical evidence synthesis (Fig. 5b). We calculated the winning rate of TrialMind versus the baseline across the five studies. The results indicate a consistent preference by annotators for the evidence synthesized by TrialMind over that of the baseline. Specifically, TrialMind achieved winning rates of 87.5%, 100%, 62.5%, 62.5%, and 81.2%, respectively. The baseline’s primary shortcoming stemmed from the initial extraction step, where GPT-4 often failed to identify the relevant sources without well-crafted prompting. Therefore, the subsequent manual post-processing was unable to rectify these initial errors.

In addition, we illustrated the ratings of TrialMind and the baseline across studies (Fig. 5c). We found TrialMind was competent as the GPT-4+Human baseline and outperformed the baseline in many scenarios. For example, TrialMind obtained the mean rating of 4.25 (95% CI 3.93-4.57) in Study #1 while the baseline obtained 3.50 (95% CI 3.13-3.87). In Study #2, TrialMind yielded 3.50 (95% CI 3.13-3.87) while the baseline yielded 1.25 (95% CI 0.93-1.57). The performance of the two methods was comparable in the remaining three studies. These results highlight TrialMind as a highly effective alternative to conventional LLM usage in evidence synthesis, streamlining data extraction and processing while maintaining the critical benefit of human oversight.

We requested that annotators self-assess their expertise level in clinical studies, classifying themselves into three categories: ‘Basic’, ‘Familiar’, and ‘Advanced’. The typical profile ranges from computer scientists at the basic level to medical doctors at the advanced level. We then analyzed the ratings given to both methods across these varying expertise levels (Fig. 5d). We consistently observed higher ratings for TrialMind than the baseline across all groups. Annotators with basic knowledge tended to provide more conservative ratings, while those with more advanced expertise offered a wider range of evaluations. For instance, the ‘Basic’ group provided average ratings of 3.67 (95% CI 3.34-3.39) for TrialMind compared to 3.22 (95% CI 2.79-3.66) for the baseline. The ‘Advanced’ group rated TrialMind at an average of 3.40 (95% CI 3.16-3.64) and the baseline at 3.07 (95% CI 2.75-3.39).

We conducted user studies to compare the quality and time efficiency between purely manual efforts and human-AI collaboration using TrialMind. Two participants were involved in both study screening and data extraction tasks. For the screening task, each participant was assigned 4 systematic review papers, with 100 candidate citations identified for each review. The participants were asked to select the 10 most likely relevant citations from the candidate pool. Each participant was provided with 2 candidate sets pre-ranked by TrialMind and 2 unranked sets. The participants also recorded the time taken to complete the screening process for each set. For the data extraction task, each participant was given 10 clinical studies. They manually extracted the target information for 5 of these studies. For the other 5, TrialMind was first used to perform an initial extraction, and the participants were required to verify and correct the extracted results. The time taken for the extraction process was reported for each study.

In Fig. 5e, we present the average performance and time cost for the AI+Human and Human-only approaches across both the study screening and data extraction tasks. The results demonstrate that the AI+Human approach consistently outperforms the Human-only approach. For the screening tasks, AI+Human achieved a 71.4% relative improvement in Recall, while reducing time by 44.2% compared to the Human-only arm. This underscores the significant advantage of TrialMind in accelerating the study screening process while also improving its quality. Similarly, for the data extraction tasks, the AI+Human approach improved extraction accuracy by 23.5% on average, with a 63.4% reduction in time required.

Detailed results of screening time and performance are shown in Fig. 5f, where two reviews showed the AI+Human approach achieving the same Recall as the Human-only arm with notable time savings, and in two other reviews, AI+Human achieved higher Recall with less time. From Fig. 5g, we see that the AI+Human approach delivered better or comparable accuracy across all three types of data, with the smallest gap in “Study design”. This is likely because study design information is often readily available in the study abstract, making it relatively easier for humans to extract. In contrast, the other two data types are embedded deeper within the main content, which can sometimes make it challenging for human readers to locate the correct information.

The overall flowchart for the study identification and screening process in building TrialReviewBench is illustrated in Extended Fig. 2.

Prompting steers LLMs to conduct the target task without training the underlying LLMs. TrialMind proceeds clinical evidence synthesis in multiple steps associated with a series of prompting techniques.

All experiments were run in Python v.3.9. Detailed software versions are: pandas v2.2.2; numpy v1.26.4; scipy v1.13.0; scikit-learn v1.4.1.post1; openai v1.23.6; langchain v0.1.16; boto3 v1.34.94; pypdf v4.2.0; lxml v5.2.1 and chromadb v0.5.0 with Python v.3.9.