QuaLLM-Health: An Adaptation of an LLM-Based Framework for Quantitative Data Extraction from Online Health Discussions

The increasing volume of health-related discussions on social media platforms, such as Reddit, presents a unique opportunity to derive meaningful insights about patient experiences, medication effects, and public health concerns.¹ However, transforming these unstructured discussions into actionable data poses significant challenges. In this study, we present QuaLLM-Health, an adaptation of the QuaLLM framework specifically designed to extract clinically relevant quantitative data from Reddit conversations about glucagon-like peptide-1 (GLP-1) receptor agonists.² This approach integrates structured prompt engineering, human-in-the-loop validation, and a comprehensive evaluation methodology to effectively translate social media content into a rich source of information for healthcare research.

Our goal with QuaLLM-Health is to leverage the strengths of large language models (LLMs) for the analysis of unstructured online forum data, enabling the extraction of actionable quantitative insights in an efficient and cost-effective manner. By integrating expert knowledge throughout the process, this framework ensures accuracy, consistency, and applicability to the healthcare domain, offering a blueprint for future research in health informatics and patient-centered studies.

We developed an annotation guideline describing variables that could be extracted as binary or categorical variables. This was used to label the textual entries (posts or comments) in a more consistent and clear way. This approach aimed to maintain consistency and clarity in identifying clinically relevant information. The annotation guideline, data used, and code are available at https://github.com/ramezkouzy/GLP1-LLM.  The guideline included the following key areas for annotation:

• •

  Inclusion and Exclusion Criteria: Posts were categorized based on whether they discussed GLP-1 receptor agonists in the context of cancer. Instances that did not meet the inclusion criteria were labeled with specific exclusion reasons to maintain dataset relevance and clarity.

• •

  Cancer Survivorship and Related Details: We identified mentions of cancer survivors, including whether they were currently taking GLP-1 medications and if they are taking it to lose weight.

• •

  Family Cancer History and Cancer Type: Posts mentioning a family history of cancer or specifying types of cancer were annotated to facilitate deeper analysis of user backgrounds and cancer types involved. Unusual or unlisted types of cancer were captured with a free-text entry.

• •

  Cancer Risk and Discussions with Physicians: Mentions related to cancer risk, including concerns, discussions of increased or decreased risk, and whether these concerns were addressed with healthcare professionals.

• •

  Timing of Cancer Diagnosis: Posts mentioning a cancer diagnosis that occurred after starting GLP-1 medication were flagged to help identify any temporal relationships discussed by users.

• •

  Information Seeking: Instances where users explicitly sought information regarding cancer risks associated with GLP-1 medications were annotated to understand user awareness and areas where more education might be needed.

This systematic and structured annotation allowed us to create a high-quality, gold-standard dataset that could reliably be used to evaluate LLM performance and guide prompt engineering. The finalized annotation guideline is attached in the published github library.^†††https://github.com/ramezkouzy/GLP1-LLM.

Two annotators with domain expertise were tasked with independently annotating the same random sample of 100 entries from the cleaned dataset. Annotations were recorded in a standardized format, capturing all variables as per the guideline. Post-annotation, discrepancies were adjudicated by discussion. The variables and the frequencies are presented in Table 1. Inter-annotator agreement was quantified using Fleiss’ kappa, with results for each variable as follows:

• •

  Variables with High Agreement ($\kappa\geq 0.8$): Discussions about GLP-1 receptor agonists in the context of cancer, mentions of being a cancer survivor, types of cancer mentioned, cases where a cancer survivor is also taking GLP-1 medication, mentions of family cancer history, discussions of cancer risk with a physician, mentions of other cancer types not previously listed, and discussions about GLP-1 potentially decreasing cancer risk. These variables showed strong consensus among annotators, indicating reliable extraction for these key areas.

• •

  Variables with Moderate Agreement ($0.6\leq\kappa<0.8$): Mentions of weight loss among cancer survivors, cancer diagnosis after starting GLP-1 medication, seeking information about cancer risks, mentions of increased cancer risk, and expressions of concern regarding cancer risk. These variables had moderate agreement, suggesting a reasonable level of consistency but room for refinement in the annotation guidelines or further training.

• •

  Variables with Low Agreement ($\kappa<0.6$): Ability to assess misinformation, overall sentiment of the post, tone of the discussion, general context, and references to misinformation. These variables showed significant variability in interpretation between annotators and were excluded from LLM performance evaluations, though they may provide useful insights in qualitative analyses.

We applied the optimized prompt to the full dataset of 2,059 unique entries. The LLM successfully extracted variables required for downstream analysis in an efficient manner, with the entire process—including prompt engineering—costing under $3 and taking around one hour to complete. The results will be presented in future analysis.

In this report, we present an adaptation of the QuaLLM framework that leverages the power of LLMs to efficiently analyze large volumes of unstructured text data, with a particular focus on extracting quantitative data. This adapted framework demonstrates value for similar and broader use cases where both quantitative and structured data are required for research purposes.

In this adaptation, we build on previously described concepts, enhancing the utility of LLMs for use cases such as healthcare-related research. We employed iterative prompting, which has proven to be highly effective, as well as few-shot prompting, both of which improved model performance.^5,6 We also emphasize the importance of involving humans at every stage of the workflow, especially in downstream deployment. Insights from domain experts play a crucial role in maximizing performance by addressing domain-specific nuances and challenges.

In addition to enhancing LLM utility, our framework exemplifies a cost-effective and time-efficient method for large-scale text analysis. The structured approach adopted in this study underscores the potential of LLMs in producing reliable and reproducible results even in complex domains such as healthcare. The combination of human expertise and iterative model tuning facilitated a nuanced understanding of social media data, which is often noisy and context-specific. This methodology not only highlights the adaptability of LLMs but also serves as a guide for future research that aims to leverage large language models for extracting actionable insights from unstructured datasets. Moreover, this approach saves significant resources and personnel time, reducing the need for manual data extraction and analysis. Future iterations of this framework could allow even more researchers to find and adapt the methodology to address their specific use cases, further expanding the potential applications across various domains.

This study demonstrates the feasibility of using LLMs for extracting clinically relevant data from unstructured social media content. The approach was cost-effective and computationally efficient, enabling a research team to conduct a study faster and with fewer resources while maintaining good accuracy and reliability.