RACER: An LLM-powered Methodology for Scalable Analysis of
Semi-structured Mental Health Interviews

Healthcare professionals and trainees across different specialties and career stages were recruited via snowball sampling method [7], described as follows. The investigators asked colleagues if they knew of anyone willing to participate in interviews about their COVID-19 experiences. Announcements were also posted online and through professional networks. Participation was voluntary with no compensation provided. Approval was obtained from the Baylor College of Medicine (Houston, TX) Institutional Review Board. The interviews were performed by a team of two research coordinators with healthcare backgrounds, and a third-year medical student, under the supervision of the investigators.

The study population of healthcare professionals and trainees consisted of 93 subjects (51 male, 42 female) with diverse demographics (Table 1). Subjects were from 22 years to over 61 years in age, and were located predominantly in Texas. Over half were married and had children. Most (75%) had no care-taking responsibilities in addition to child-care. Professionally, the sample included physicians (54.8%), medical students (21.5%), nurses (8.6%), residents (7.5%) and other healthcare professionals. Subjects trained at multiple institutions, with prominent representation from Baylor College of Medicine and University of Texas systems. Various specialties were represented in the cohort, with emergency medicine, psychiatry and pulmonary/critical care among the most common.

SSIs were conducted over videoconferencing using a standard template consisting of a total of 41 questions, including four questions that were only asked to students, and seven questions that were asked to only non-students. Questions were either factual, concerning demographics and personal and professional background, or open ended, where interviewees were asked to talk about their experiences during the COVID-19 pandemic, focusing on their exposure to the virus, work impacts, emotional responses, future outlooks, and coping strategies. Interviewees discussed how they had practiced in high-risk areas, their concerns for personal and family safety, and modifications made to their routines. They also reflected on the physical toll the crisis had taken. The impact on their work included changes in working hours, shifts in patient care quality, and altered management approaches. Emotional and psychological questions revealed how the crisis affected them emotionally, the level of support they received, family dynamics, and changes in burnout levels. Looking ahead, they pondered the crisis’s short-term and long-term impacts on their careers and specialty choices. Finally, they shared their openness to seeking help for burnout or mental overwhelm and identified potential obstacles in obtaining this help. Students were not asked clinical-practice related questions, and were instead asked about how their training was being affected by pandemic-related changes. Interviews lasted on average 26.7 +/- 8.9 s.d. minutes. When transcribed from raw interview audio into text transcripts (using Otter.AI[8]), were on average 4044.30 +/- 1348.34 s.d. words long.

We developed an LLM-based automated pipeline called RACER (Figure 1) that converts a corpus of text SSI transcripts into insightful themes per interview question. RACER, consists of four stages, Retrieve, Aggregate, Cluster with Expert guidance, and Recluster:

Retrieve: We first structured interview transcripts by using an LLM (OpenAI’s GPT-4[2]) to retrieve relevant SSI text in response to each of the questions in the interview template. (See Methods LLM prompt details) To avoid LLM ‘hallucinations’ [9], we asked the LLM to provide ‘evidence’ in the form of text quoted verbatim from the transcript, to back up its response to each question. LLM outputs missing either answers or backing evidence to any question were automatically detected and rerun.

Aggregate: For each question, we then aggregated the retrieved responses across all subjects who were asked that question.

Cluster with Expert guidance: We then asked the LLM to semantically cluster the responses into primary and secondary clusters (‘themes’ and ‘sub-themes’). For most questions, we provided the LLM expert-guidance in the form of primary-cluster definitions. The LLM discovered secondary clusters (or sub-themes) automatically. Expert-provided cluster definitions were always mutually exclusive and collectively exhaustive, while those discovered by the LLM were not constrained to be so. Similar to before, invalid LLM responses, e.g. those missing cluster assignments for any subjects, were automatically re-run.

Re-Cluster: Leveraging the probabilistic nature of LLMs, we assessed the robustness of the clustering process by re-running it four more times, employing the same cluster definitions and validation criteria as in the initial step. We used a majority vote over 5 runs to assign subjects to clusters, to get robust cluster assignments for all downstream processing. The number of votes (3, 4 or 5 out of total 5 LLM calls) additionally provided a synthetic measure of LLM confidence [10, 11]. Only a very small fraction of subject-question pairs (12 out of 3342, 0.36%) had no ‘robust’ cluster assignments after applying the majority voting process.

All together, we found that RACER was able to take unstructured transcriptions and extract relevant and insightful, clustered responses in a robust manner for downstream human analysis.

To validate the output of running RACER on our SSI dataset, two human evaluators cross-checked the resulting cluster assignments for 20 randomly-selected subjects across 28 open ended questions. Using the same cluster definitions as were previously used by RACER, each human evaluator (E1 and E2) independently read the raw transcript file and assigned each subject’s answers to the primary clusters. Evaluator cluster assignments were then compared to RACER’s robust cluster assignments. To quantify agreement, we defined a concordance score and a concordance ratio as follows: If the clusters for a given subject-question pair matched exactly (for mutually exclusive clusters), or were sub-sets or super-sets of each other (for mutually non-exclusive clusters) they were assigned a concordance score of 1. Conversely, mismatch was assigned a concordance score of 0. The overall concordance ratio is the proportion of matched subject-question pairs between evaluators.

We observed a concordance of 78% (E1) and 87% (E2) between human evaluators and RACER, and a 77% (E1-E2) inter-rater concordance (Figure 2B). When all three evaluators were compared simultaneously, there was a little decrease in the concordance (72%), indicating that across the majority of subject-question pairings, cluster assignments produced by humans and RACER were all in agreement. See Methods for additional details.

We examined the confidence score produced by RACER per subject-question pair to see how it might affect the subject-question pair’s concordance with human evaluators (Figure 3). Amongst the 443 subject-question pair sample evaluated by humans, 392 (87.7%) were entirely robust (5 of 5 repeat clusters). This was similar to the population confidence score distribution 88.2% (1852 of 2099 subject-question pairs). RACER’s average confidence across all subjects for a given question showed significant and positive correlations with Human-RACER concordance of the 20 subjects evaluated for those same questions. Additionally, we observed that the confidence scores of subject-question pairs that were concordant between human evaluators and RACER, were higher than for non-concordant subject-question pairs. This was due to significant differences in the proportion of lower confidence subject-question pairs between concordant and non-concordant groups. Interestingly, when we juxtaposed RACER confidence scores against human-human concordance, we observed that RACER confidence was lower when humans were non-concordant. This suggests that areas where RACER was less confident or ‘confused’ were also areas where human evaluators tended to disagree. Thus the RACER confidence generated via repeated clustering could also serve as an indicator of SSI ambiguity or difficulty.

We now summarize the insights gleaned from analyzing SSIs with 93 subjects using our automated processing pipeline.

Our study demonstrates the utility of RACER for efficiently analyzing semi-structured interviews (SSIs), particularly those exploring complex mental health topics within the healthcare domain. We introduce a novel approach by employing RACER to analyze emotions and psychological behaviors, opening new possibilities for exploration in mental health. By providing expert-guided constraints and using automated response validation steps, RACER accurately extracts and robustly clusters relevant responses from interview transcripts. Automating these laborious manual tasks significantly enhances the scalability of SSI analysis. The inter-rater agreement between LLM-assigned clusters and human expert clusters further bolsters our claims. The automated pipeline achieved moderately high concordance compared with manual evaluation by human annotators. The overall concordance ratio of 0.72 for RACER versus both human evaluators approaches the 0.77 concordance ratio between the two human evaluators.

Our findings reveal both the promises and current pitfalls of LLMs for SSI analysis. We found that when the RACER struggled with robust clustering, both humans and machine were more likely to be non-concordant which could suggest shared limitations in handling complex emotions or psychologically nuanced statements [12] or ambiguity of the underlying SSI. This underscores the indispensable role of human expertise in reviewing and interpreting LLM outputs, where RACER’s confidence levels can guide expert scrutiny.

While RACER provided evidence in the form of quoting relevant interview text to support its response in the Retrieval step, the underlying methodology remains opaque. In contrast, human evaluators were able to describe their techniques, even if subjective. For instance, humans considered different amounts of contextual information outside the question scope, and inferred subject intentions to varying degrees, i.e. whether the subject needed to explicitly say certain phrases, or if they could be inferred from previous statements or knowledge of the subject matter. An LLM’s ability to consider large amount of contextual information can be a double-edged sword; beneficial if relevant information appears elsewhere in the transcript, but misleading if the research is indeed directed towards a narrow window of text around the question.

We demonstrated that LLMs can help discover knowledge by automatically extracting themes and topics from subject responses. However, good performance requires clear, mutually exclusive category definitions. We found it highly useful to involve domain experts early to precisely define mutually exclusive thematic clusters. For certain questions, where succinct mutually exclusive categorization was not possible, we chose to use LLM-discovered clusters. However, validation of such non-exclusive categorization is challenging. Our results showed higher LLM accuracy and inter-rater agreement for questions with non-overlapping expert-defined clusters versus those allowing multiple clusters.

Additionally, human evaluators exhibited biases, such as default cluster tendencies requiring countering evidence (e.g. starting from a default of ‘no’ and requiring evidence to switch to a ‘yes’, or vice versa). Thus, expert human analysis also demonstrates cognitive variability and individual biases. Rather than definitive classifications, both human and machine outputs should be considered informed yet inherently biased perspectives on complex qualitative responses [13]. Thus, in the future, clearly delineating the parameters of evaluations with humans and RACER may improve concordance.

While RACER’s cluster assignments may deviate slightly from human reviewers, RACER was internally consistent and demonstrated high clustering repeatability for most questions. Furthermore, unlike humans, RACER was able to efficiently process an extensive dataset of 93 subjects and can scale to significantly larger data set sizes that would otherwise be infeasible for human evaluators to handle.

For researchers undertaking projects in this emerging domain, both optimism and caution are warranted [14, 15, 16, 17, 18, 19]. With appropriate constraints and validation, LLMs can accelerate knowledge extraction from SSIs. We implemented safeguards against hallucination risks like requiring verbatim textual evidence for an answer, which constrained the LLM to mostly avoid fabricating content. While this is already an area of active research, the possibility of a few false positives remains and needs to be accounted for in downstream use.

While evaluation of LLM outputs through comparison to multiple human raters was helpful through comparison to multiple human raters, inter-rater agreement must also be looked at to assess inherent ambiguity. To further improve performance, we recommend specialized training for both SSI interviewers and human evaluators.

We found it useful to generate an ensemble of LLM clustering outputs from repeated runs, and used it to extract robust cluster assignments and to get a measure of model uncertainty. Future work exploring this direction could produce useful methods that help build trust in LLM-assisted analyses and inform human-in-the-loop processes for high-stakes applications [20].

Interviewers were provided with a standard template to guide their discussions. The subjects were all healthcare professionals or trainees, including physicians, nurses, and medical students. The interviews followed a semi-structured format, where the interviewers were instructed to cover a previously decided list of questions, and were allowed to ask exploration questions if the ‘root’ question was not answered. The questions covered in the SSIs are listed in A. Raw audio / video interview files were transcribed into text format using the Otter.AI transcription service [8]. Out of 100 interviews conducted, 7 were compromised due to data-corruption/loss issues, providing a total of 93 transcriptions for further processing. Voice to text transcription was carried out using Otter.AI[8], which attempts to perform automated speaker diarization, but does not do so perfectly. To the best of our knowledge, this shortcoming did not seem to influence the subsequent processing steps.

We used the OpenAI GPT-4 LLM for all our work, except for prompts which exceeded GPT-4’s limits, where we used GPT-4-32k.

Our study employed human evaluation to verify the alignment between RACER-generated clusters and human interpretation, utilizing two independent evaluators who analyzed the responses of 20 randomly selected subjects from a pool of 93. Each evaluator individually reviewed the raw interview transcript files for the selected 20 subjects and used the same cluster definitions as RACER to assign subjects to clusters. Human evaluators spent approximately 30 minutes per subject on average for a comprehensive review and categorization of the responses. This time investment reflects the thoroughness and attention to detail applied by the evaluators in their analysis, and also highlights the limits of this process to scale to large study populations. To validate the semantic clustering results produced by the LLM, each human evaluator compared their assigned scores with those generated by the LLM. An inter-rater comparison was also conducted, involving a detailed examination of the scores and evaluations independently made by both human evaluators (E1 and E2) for the same set of subjects. Concordance scores of 1 were assigned to clusters that precisely matched or were sub- or super-sets of each other, while discrepancies received a concordance score of 0. The overall concordance ratio represented the proportion of clusters aligning between the evaluators.

Additionally, the evaluators’ findings were juxtaposed with RACER’s cluster assignments to gauge both inter-evaluator consistency and the degree of correspondence with the LLM’s outcomes. We also compared the use of Cohen’s kappa coefficient with our concordance score and found them to be similar (Appendix 8). Due to the nature of the comparison across questions which varied in the number of possible clusters as well as probability of different cluster assignment across questions, the concordance scores were used as they better described the intended comparisons. Instances where RACER did not produce any robust cluster assignments were categorized as ’mismatch’ during the evaluation process.