Automating PTSD Diagnostics in Clinical Interviews:
Leveraging Large Language Models for Trauma Assessments

The advent of LLMs like GPT (OpenAI, 2023), Llama (Touvron et al., 2023), and PaLM (Chowdhery et al., 2022) has sparked research into their applications in mental health (Ji et al., 2023). One key area is using conversational agents for mental health support and counseling, where LLMs excel at generating empathetic responses (Lai et al., 2023; Ma et al., 2023b; Loh and Raamkumar, 2023), highlighting their potential as digital companions or on-demand service providers. Additionally, the research on decision-support systems for novice counselors underscores their potential to enhance mental healthcare provision (Fu et al., 2023).

Research has also explored LLMs in disease detection and diagnosis (Zhang et al., 2022), focusing on issues like depression (Qin et al., 2023), stress (Lamichhane, 2023), and suicidality (Bhaumik et al., 2023). Closer to our work, Bartal et al. (2023) use text-based narratives from new mothers to assess childbirth-related PTSD with GPT and neural network models. Although GPT showed moderateperformance, it holds promise for clinical diagnosis with further refinement. These studies typically use zero/few-shot prompting for binary or multi-label classification, demonstrating LLMs’ capabilities in detecting mental health issues without fine-tuning, despite challenges like unstable responses, potential bias, and interpretation inaccuracies.

Some research has pivoted towards fine-tuning LLMs for domain-specific performance enhancement. Xu et al. (2023) present two fine-tuned models, Mental-Alpaca and Mental-FLAN-T5, outperforming GPT-3.5 and GPT-4 in multiple mental health prediction tasks. Based on Llama-2, Yang et al. (2023) train MentaLLaMA on 105K social media data enhanced by GPT. The model performance is on par with other state-of-the-art methods, while providing interpretable analysis.

Research on using LLMs on clinical interview data and diagnosis is limited. Wu et al. (2023) utilize GPT to augment the Extended Distress Analysis Interview Corpus by generating a new dataset from provided profile and rephrasing existing data. The augmented data outperforms the original imbalanced data in PTSD diagnosis. Galatzer-Levy et al. (2023) adopt Med-PaLM-2 to predict Major Depression Disorder (MDD) and PTSD on eight item Patient Health Questionnaire and PTSD Checklist-Civilian version ratings.

Participants were paid $60.00 for this interview and underwent semi-structured diagnostic interviews conducted by doctoral-level clinicians or doctoral students supervised by a licensed clinical psychologist on staff. A total of 411 interviews were conducted with 336 unique participants, some of whom had follow-up interviews after >1 month. 93.4% ofthe participants were women and 79.5% were Black or African American ($M_{age}$ = 31.4), where 38.7% had a high school education or less and 57.9% reported a monthly household income of < $1,000.

The diagnostic interview begins with a section of the Longitudinal Interval Follow-Up Evaluation to assess global adaptive functioning across various psychosocial domains, including work, household, relationship as well as general functioning, and life satisfaction in the past month Keller et al. (1987). Videos of the interviews are recorded using online conferencing software such as Zoom and Microsoft Teams. Each interview lasts 1.5 hours on average, involving the participant and 1-2 interviewers.

A total of 10 sections are applied during the interview. Among them, 4 sections are administered to the majority of participants; thus, this study focuses on those 4 sections. The first two sections, the Life Base Interview (LBI) and the Treatment History & Health (THH), are internally designed to assess the history of psychiatric diagnoses and treatment, as well as the presence of suicidality. The other two sections, the Criterion A (CRA) and the Clinician-Administered PTSD Scale for DSM-5 (CAP), follow the standard diagnostic criteria for PTSD outlined in the Diagnostic and Statistical Manual of Mental Disorders (DSM-5; Weathers et al. (2018)). Everysection is accompanied by a set of questions, linked to variables that store pertinent values derived from the corresponding answers. Table 1 shows statistics and examples for each of the 4 sections.²²2Descriptions of all 10 sections are provided in Appendix A.

Two commercial tools, Rev AI³³3Rev AI: https://www.rev.ai and Azure Speech-to-Text⁴⁴4Azure Speech-to-Text: https://bit.ly/42r24pA, and an open-source tool, OpenAI Whisper Radford et al. (2023), are tested for automatic speech recognition (ASR) on our dataset. Whisper gives the lowest Word Error Rate (WER; Klakow and Peters (2002)) of 0.13, compared to 0.21 and 0.16 from Rev AI and Azure, respectively. Whisper also exhibits better performance in handling noisy environments and numbers that Azure often misses or inaccurately transcribes (Table 2). Despite its superior ASR performance, Whisper does not identify speakers, a feature found in the others. Thus, both Azure and Whisper are run on all audios and their results are combined to obtain the best outcomes.

To map the speaker diarization (SD) output from Azure to the Whisper output, Align4D⁵⁵5Align4D: https://github.com/emorynlp/align4d is used such that the first and last words of every utterance in the Azure output are aligned to their corresponding words in the Whisper transcript with speaker info, and form a speaker turn spanning all words between those words. Some words in the Whisper transcript may get left out from this mapping, which are combined with either preceding or following adjacent utterances using heuristics.

Text-based Diarization Error Rate (TDER; Gong et al. (2023)) is used, more suitable than traditional metrics like WER or Diarization Error Rate (DER; Fiscus et al. (2006)), for evaluating text-based SD. Transcripts from 29 audios produced by Microsoft Teams are used as the gold-standard, where Teams identifies speakers via different audio channels with near-perfect SD. Our aligned method achieves a TDER of 0.56, a significant improvement over the TDER of 0.62 achieved by Azure alone.

Each interview is conducted through multiple sections comprising a series of questions (Section 3.3), yet recorded as one continuous video. It is crucial to segment the video into sections, each of which is split into sessions, where a session contains content relevant to a specific question. Here, a session is defined as a list of utterances where the first utterance includes the corresponding question, and it is followed by another session whose first utterance includes the next question (if it exists). Algorithm 1 describes how a section is matched in the transcript.

Finally, Algorithm 3 shows how session spans are found for a specific section. $C^{e}$ is a list of tuples comprising utterance IDs and their scores for the $k$’th section created by Algorithms 1 and 2 (L1) (ro: remove_overlap). $C^{\ell}$ is created in the same manner, except adapting the Levenshtein Distance (LD) as the similarity metric (L2) Levenshtein (1966). $\texttt{sel}(C,s)$ returns a list of tuples comprising utterance IDs and their matched question IDs, where the scores > $s$. $\texttt{last}(U,Q^{c}_{*})$ returns the first utterance ID of the $(k+1)$’th section if exists; otherwise, it returns the last utterance ID of $U$. $C$ is created by taking the intersection of $C^{e}$ and $C^{\ell}$ whose scores > 0.8 and 0.7, and the last utterance ID (L3).⁷⁷7Any section not matched by Algorithm 1 is considered absent.

For each span $U^{\prime}$ of utterances between $C_{i}$ and $C_{i+1}$ (exclusive for both ends), a list $Q^{\prime}$ of optional questions related to $C_{i}$ is created (L5-7). $T^{e}$ is a list of tuples comprising utterance IDs in $U^{\prime}$ and their matched question IDs in $Q^{\prime}$ with scores > 0.8, and $T^{\ell}$ is created using LD (L8-9). The intersection of $T^{e}$ and $T^{\ell}$ is appended to a list $O$ (L10), which is then merged with $C$ and sorted to produce $V$ (L11).

For each span $U^{\prime\prime}$ between $V_{i}$ and $V_{i+1}$, a list $Q^{\prime\prime}$ of any questions have not been matched in that spanis created (L14). Bipartite matching bw. $U^{\prime\prime}$ and $Q^{\prime\prime}$are performed to find matches optimizing severalcriteria in Appendix B.1 (L15), accumulated, merged, and sorted to produce the final list (L16-17).

Answers to the questions are used to determine the values of the variables (Table 1), resulting in many-to-many relations between questions and variables (many-questions to one-variable is the most common case). Our data comprises five variable types. (1) Scale assesses on an ordinal scale with ratings for intensity, severity, or likeness. (2) Category selects among binary choices or distinct class labels. (3) Measure captures various units such as duration, frequencies, and ages. (4) Notes are summarized texts documented by the interviewers. (5) Rule is calculated based on predefined rules derived from the other variable types. Table 3 shows the statistics of all variables for each section in our dataset.

The state-of-the-art commercial and open-source large language models, GPT-4 and Llama-2 Touvron et al. (2023), are adapted for our experiments.⁹⁹9Specific versions, parameters, and costs for these large language models are provided in Appendix C.3 and C.4. For each question, a model takes all sessions related to the variable to which the question pertains (§4.4), and an instruction to provide the answer and explanation. Table 4 shows our templates including replaceable patterns to generate the instruction for each variable type. For Scale, {keywords} can be replaced with "how severe", and {symptom} with "have unwanted dreams in the past month". For Category, {keywords} can be replaced with "which of the following categories best describes", and {symptom} with "usual employment status". To constrain the answer generated by the model, details such as the answer {range} for S&C, and the value {type} for Measure are incorporated. S has a special pattern {attributes}, directing the model to return a particular score under certain conditions.

Assessing model performance for Notes poses a challenge as they must be compared against text summarized by interviewers. Given the complexity of this task, it is decomposed into multiple subtasks of binary classifications, information extraction, and categorization by adapting Chain-of-Thought Wei et al. (2023). First, GPT is asked to generate a list of slots for each N variable, based on a batch of summary notes from interviewers. Because many of these slots have similar meanings, albeit varying in naming, GPT is again asked to cluster them. The clusters generated by GPT are manually refined, resulting in final grouped slots that cover 95+% of the initial generation. For each of these slots, an LLM is tasked with determining if relevant content for the slot is present in the provided sessions.¹⁰¹⁰10Appendix C.5 gives slot examples for Notes variables.

Zero-shot and few-shot settings are tested across all variable types¹¹¹¹11Appendix C.2 gives details on zero/few-shot settings.. For Scale, two few-shot settings are explored: one including an example for a single scale point, and the other covering examples for all scale points. For the GPT model, few-shot settings mostly outperform zero-shot settings in predicting Category, Measure, and Notes variables. For Scale, the few-shot setting with a single example results in the lowest performance. On the other hand, the few-shot setting including examples for all scale points shows a slight improvement in model performance. Thus, few-shot settings are used for all experimentswith GPT. In contrast, the Llama model consistently yields inferior outcomes with few-shot settings compared to zero-shot settings, leading us to adopt zero-shot settings for all Llama experiments.

Since each variable type is uniquely defined, different evaluation metrics are employed accordingly. Accuracy is computed for all types except Notes. For Notes, since the model identifies the presence of information in the provided sessions based on predefined slots, Recall is used as the primary metric to gauge the coverage of relevant information detected by the model. For Scale, the Root Mean Square Error (RMSE) and Bias evaluation are used. RMSE quantifies the magnitude of errors, whereas Bias evaluation calculates the proportion of positive and negative residuals, thereby revealing any directional bias in the model predictions.

Table 6 gives the results for each variable type. For Scale, additional evaluation is conducted for CAP whose original scaling ranges from 0 to 4 where 0 indicates the absence of symptoms, 1 denotes minimal symptoms, and 2+ are considered symptoms that meet or exceed the threshold for clinical significance. To reflect this clinical demarcation, scale points are categorized into three scale groups, 0, 1, and 2+, and evaluated as Scale_g.¹²¹²12Appendix C.6/C.7 presents results for each section/variable.

GPT consistently shows significantly higher accuracy, averaging 10.5% more across all types than Llama, and reaches an accuracy of 68.4% for Rule accumulating outcomes of other types. Regarding RMSE, GPT exhibits an error rate of 0.8 for Rule using results of Scale, implying that it is less than one scale off from human judgment on average. In terms of Bias, ranging from -1: completely biased to negative to 1: completely biased to positive, GPT displays a marginal bias toward negative for Scale, while Llama shows a strong positive bias, implyingthat GPT is a bit conservative in predicting a higher scale, whereas Llama tends to overestimate. GPT underestimates more than Llama for Measure, however, showing a slight negative bias of 0.15 for Rule. For Notes, Llama exhibits better performance with a recall of 52.7% than GPT, suggesting that Llama is more effective in retrieving relevant information. Considering that these models are not fine-tuned onour data, this level of performance is very promising, as we can achieve a robust model for practical use with further training.