LLM-based Conversational AI Therapist for Daily Functioning Screening and Psychotherapeutic Intervention via Everyday Smart Devices

People who experience mental health adjustment issues and disorders tend to face diminished capacity in professional or academic performance, maintaining social relationships, and self-care (Helfrich et al., 2008; Bible et al., 2017). Traditional screening tools, such as the Mental Status Examination (MSE), require clinicians to observe and assess people’s daily functioning, such as physical appearance and presentation, social interaction behaviors, and emotional expression (Trzepacz and Baker, 1993). Other widely used diagnostic assessments, such as the Adult ADHD Self-Report Scale (ASRS-v1.1), the Patient Health Questionnaire-9 (PHQ-9) for depression, and the General Anxiety Disorder-7 (GAD-7), which provide more specific screening options for specific mental health diagnoses, often include questions or items assessing daily functioning (Kroenke et al., 2001; El ASRS, 2009). For example, the PHQ-9 includes assessments of mood, sleep hygeine, and eating habits (Kroenke et al., 2001).

There are several psychological measurements designed to examine the day-to-day functioning of individuals to evaluate their mental health well-being, such as DLA-20 and GAF (Nie et al., 2022; DSM-IV-TR., 2000; Scott and Presmanes, 2001). DLA, which was developed to evaluate aspects of daily functioning affected by mental illnesses, includes 20 major categories for daily functioning. These categories include interpersonal communication, family relationships, personal hygiene, time management, and productivity at work (Scott and Presmanes, 2001). On the other hand, GAF, which was introduced in DSM-IV, employs an ordinal scale to evaluate an individual’s overall level of functioning (DSM-IV-TR., 2000). A lower GAF score indicates the presence of more significant symptoms and difficulties in social, occupational, and psychological functioning.

Clinicians using evidence-based practices (EBP) in psychology to guide interventions and treatment plans, taking into account relevant research on their clinical practices, have found that the use of EBP helps improve the quality and accountability of clinical practices (on Evidence-Based Practice et al., 2006; Spring, 2007). Some commonly used EBP include CBT, acceptance and commitment therapy (ACT), DBT, and MI. CBT is one of the most popular and commonly used psychological interventions. It focuses on challenging one’s cognitive distortions and subsequent behaviors to reduce existing mental health symptoms and improve overall mental well-being (Alford et al., 1997; Beck and Beck, 2011). CBT is found to be effective in a variety of diagnoses, such as mood disorders, Attention-deficit/hyperactivity disorder (ADHD), eating disorders, Obsessive-compulsive disorder, and Post-traumatic stress disorder (Clark et al., 2003; Roy-Byrne et al., 2005; Emilsson et al., 2011; Halmi et al., 2005; Walsh et al., 2004; Foa et al., 2005; Dickstein et al., 2013). CBT also shows promising results in preventative care that may not be tied to a specific diagnosis. It has been effective in various settings, including medical, work, and school environments (Moss-Morris et al., 2013; Tan et al., 2014; Miller et al., 2011). However, despite the abundant evidence of its effectiveness, CBT is associated with a high nonresponse rate, attributed to participants’ low motivation (Antony et al., 2005). During CBT, therapists assess the validity and utility of participants’ responses to understand their thought patterns and beliefs accurately  (Sokol and Fox, 2019). Such an assessment involves identifying, challenging, and reframing cognitive distortions, such as overgeneralization, emotional reasoning, all-or-nothing thinking, catastrophizing, etc  (Burns and Beck, 1999). CBT usually consists of the following steps:

1. (1)

   Identify the Situation/Issue: Start by clearly identifying the situation or issue you want to work on.

2. (2)

   Recognize Negative Thoughts: Think about the thoughts that go through your mind when you experience this issue. These are often automatic or subconscious thoughts that may be irrational or unhelpful. They can be self-critical, overly pessimistic, or unrealistic.

3. (3)

   Challenge Negative Thoughts: Challenge means questioning the validity of these thoughts. Are there alternative, more balanced, or rational thoughts that might be more helpful in the situation?

4. (4)

   Reframe Thoughts and Situations: Try to reframe your unhelpful thoughts and situations into more balanced, realistic, and constructive ones. This process is about changing the way you think about the situation, which can lead to changes in your emotions and behaviors.

To address issues related to low motivation, researchers have suggested using MI as a complementary approach alongside CBT  (Marker and Norton, 2018; Naar and Safren, 2017; Arkowitz and Westra, 2004). There are four techniques to effectively implement MI  (Miller and Rollnick, 2012):

1. (1)

   Open-ended questions: Encouraging elaboration on responses, asking for examples, or exploring the implications of what’s been shared;

2. (2)

   Affirmations: State strengths and help feel that changes are possible;

3. (3)

   Reflective listening: (i) Simple reflection: repeating what the client has said, using slightly different words or phrases; (ii) Reframe reflection: listening to the client’s statements and then reflecting them back in a way that presents a new perspective or interpretation; and (iii) Affective reflection: recognizing, understanding, and reflecting back the emotional content of what the client expresses;

4. (4)

   Summaries: Use summaries not only to encapsulate discussions but also to highlight progress.

MI is an evidence-based practice for substance use disorders and other addiction issues (Anton et al., 2006; Aarons et al., 2017). It is also found to be effective in helping people adapt to various situations, such as managing diabetes (Kertes et al., 2011; Channon et al., 2007; Chen et al., 2012). Growing research has shown that the combination of CBT and MI shows effectiveness in a variety of populations and for mental health adjustments (Merlo et al., 2010; Marker and Norton, 2018; Kertes et al., 2011; Arkowitz and Westra, 2004).

There are various methods for mental health self-screening (Kruzan et al., 2022; Brown et al., 2016). While online help-seeking is preferred by many individuals (Gould et al., 2002), these tools provide a limited assessment based on closed-ended questions, potentially leading to omitting important details typically obtained from open-ended questions or in-person interactions (Screening, 2021; Onl, 2022). Besides the online tools, (Liu et al., 2022) proposed an AI-based self-administer online web-browser-based mental status examination (MSE). Recently, Experience Sampling Method (ESM) has been widely adopted by HCI researchers for various physical and mental health screening. ESM can be done automatically by sensors or by repeatedly prompting users to answer questions in their normal environments. For example, ESM is used for self-reporting Parkinson’s Disease symptoms, chronic pain, designing health technologies for Bipolar Disorder, etc (Vega et al., 2018; Adams et al., 2018; Matthews et al., 2015). Most wellness self-screening methods in the literature use close-ended questions and expect close-end results from the user, while CaiTI uses open-ended questions and allows the user to chat freely with any topic. And they usually only focus on particular dimensions of the day-to-day functioning or mental disorders instead of performing a comprehensive screening.

Large Language Models (LLMs) are pre-trained on vast datasets, which equip them with significant prior knowledge and enhanced reasoning skills. The recent state-of-the-art models, including GPT-4 (OpenAI, 2023b), GPT-3 (OpenAI, 2023a), Claude-3 (Anthropic, 2024), and Gemini 1.5 (Pichai and Hassabis, 2024), exhibt strong capability in reasoning over text. Consequently, recent research increasingly employs LLMs alongside various language, vision, or multimodal models to enable advanced applications in various domains without the need for additional training (Yin et al., 2023; Sharan et al., 2023; Deb et al., 2023). For example, IdealGPT combines two LLMs (GPT) with a vision-and-language model to enable a framework that iteratively decomposes vision-and-language reasoning, where the two LLMs are treated as Questioner and Reasoner (You et al., 2023). Additionally, research has shown that LLMs possess the ability to reason with and interpret IoT sensor data (Xu et al., 2023a).

Recently, transformer-based Large language foundation models, such as GPT-4 (Bubeck et al., 2023), PaLM 2 (Anil et al., 2023), and LLaMA2 (Touvron et al., 2023), have demonstrated superior performance across various medical-related NLP tasks. LLMs are used to enable various general healthcare applications. (Waisberg et al., 2023) showed that GPT-4 has the potential to help drive medical innovation, from aiding with patient discharge notes, summarizing recent clinical trials, and providing information on ethical guidelines. Moreover, Google introduced Med-PaLM and Med-PaLM 2(Singhal et al., 2023a, b), LLMs explicitly tailored for the medical domain, providing high-quality responses to medical inquiries.

Various works also exploit and evaluate the performance of LLMs for mental status classification and assessment. Researchers leveraged LLMs for mental health prediction via online text data and evaluated the capabilities of multiple LLMs on various mental health prediction tasks via online text data (Xu et al., 2023b; Radwan et al., 2024). In addition, (Jiang et al., 2023) leverages RoBERTa (Liu et al., 2019) and Llama-65b  (Touvron et al., 2023) in the system for classifying psychiatric disorder, major depressive disorder, self-rated depression, and self-rated anxiety based on time-series multimodal features.

In addition to assessing and classifying the mental status of the user, researchers have investigated providing psychological consultations. For example, (Nie et al., 2022) leveraged GPT-3 to construct a home-based AI therapist that detects abnormalities in mental status and daily functioning and generates responses to console users. (Lai et al., 2023) proposed an AI-based assistive tool leveraging the WenZhong model, a pre-trained model trained on a Chinese corpus for question-answering in psychological consultation settings (Lai et al., 2023). Researchers also investigate the potential of ChatGPT in powering chatbots to simulate the conversations between psychiatrists and mentally disordered patients (Chen et al., 2023).

Only a few works in the literature focus on using LLM for MI or CBT, developing these psychotherapy systems, and evaluating them in real-world scenarios. (Kian et al., 2024) developed a GPT3.5-powered prompt-engineered socially assistive robot (SAR) that guides participants through interactive CBT at-home exercises. Their findings suggest that SAR-guided LLM-powered CBT may yield comparable effectiveness to traditional worksheet methods. However, this study solely focused on employing an LLM-based approach to simulate traditional worksheet-based CBT, without thoroughly examining the validity of user responses to the CBT exercises or ensuring users effectively engaged with the CBT.

Converse with the User in a Natural Way and the Psychotherapeutic Conversational Intervention modules of CaiTI are closely related to each other when CaiTI converses with the user. Based on the psychotherapists’ experience in dealing with thousands of clients, several factors are considered to shape the conversation process of CaiTI. First of all, when the therapist asks a question, some clients express a lot, while others do not respond to the question, but talk about other things (related to other dimensions). In addition, not all clients are patient enough to go through all dimensions that the therapist wants to check. Psychotherapists usually start to check on the dimensions that the clients didn’t do well in previous sessions and are more important for assessment. If clients have a problem in a dimension, the therapists usually follow up to hear more about this dimension and provide quick counseling and therapy addressing the specific issue. This mirrors the psychotherapist’s tendency to focus on one problematic dimension extensively rather than treating multiple dimensions at once.

Taking the professional experiences and common practices of the psychotherapists into consideration, to converse with the user in an efficient, intelligent, and natural way to screen physical and mental health status, CaiTI’s conversation process follows four guidelines:

1. (1)

   Prioritize questions intelligently: CaiTI starts with the dimensions that concern therapists more, while personalizing the priority to each user and formulating questions based on his/her historical responses.

2. (2)

   Understand the user input better: CaiTI checks if the user answers the question asked (Dimension_N), understands how well the user performs in this dimension and decides if follow-up questions and conversational interventions are needed.

3. (3)

   Obtain more information through minimal questioning: CaiTI maps each user response to all possible dimensions to avoid redundant questions.

4. (4)

   Guarantee the quality of psychotherapies: CaiTI intelligently interprets and reasons the user responses and, when needed, it steers and guides the conversation back to the psychotherapeutic goals if the user’s answers stray.

Figure 4 shows CaiTI’s process to converse with the user. Generally, MI therapies are conducted throughout the conversation, while CBT proceeds at the end of the conversation session. There are four modules in which the LLMs are involved: CaiTI Questioner, Response Analyzer, reflection-validation (R-V) process, and CBT process.

In particular, CaiTI asks one question for each dimension if CaiTI does not obtain any information in the dimension from the user’s previous responses. A model-free reinforcement learning algorithm, Q-learning, is used to decide the action (i.e., the next question) in the current state (i.e., the current question). For each dimension (Dimension_N), CaiTI Questioner formulates the question and uses the text-to-speech method to converse with the user through the front-end device. The front-end device generates the text of the user response (speech-to-text conversion is used if the user has voice input). The detailed implementation of the front-end device is described in Section 6.3.

CaiTI expects the user to chat freely with it and can deal with open-ended responses. CaiTI performs segmentation on the user response in to $M$ Segment(s). For each Segment, a LLM-based Response Analyzer, described in Section 5.2, is used to classify the Segment into (Dimension, Score). If CaiTI fails to classify the Segment into the format of (Dimension, Score), it asks the user to rephrase the answer. CaiTI logs the user response if CaiTI still fails to classify the rephrased Segment into the format of (Dimension, Score). Otherwise, CaiTI checks the Score no matter if this Segment is answering the question asked by CaiTI or not. If the user needs more attention in this dimension (Score $=2$), CaiTI proceeds with a reflection-validation (R-V) process starting with asking for more information starting with a simple reflection in MI. An example of this process is presented in Figure 5.

The R-V process is described in detail in Section 5.3 and demonstrated in Figure 8. To ensure the user provides follow-up information in the right direction, an R-V Reasoner and an R-V Guide are deployed. Based on the user response to the original and follow-up question, CaiTI provides validation, which includes affective reflection and affirmations in MI. Then, CaiTI assigns the Score to Dimension. After handling all Segment(s), CaiTI verifies whether the user does respond to the question asked by CaiTI in Dimension_N. In cases where a user does not answer the question asked by CaiTI (Dimension_N) but talks about something else, CaiTI asks the question in Dimension_N again.

After CaiTI enumerates all dimensions or the user wants to stop the session, CaiTI provides a summary of the chat session and asks the user to choose a dimension to work on for the CBT process. This CBT process includes the four steps outlined in Section 2.2. In particular, CaiTI identifies the situation and issue in the dimension the user chose based on the conversation history. Then, CaiTI leads the user to recognize (CBT Stage_1), challenge (CBT Stage_2), and reframe (CBT Stage_3) the negative thoughts in this situation. To ensure the effectiveness and quality of the CBT process, each CBT stage contains a Reasoner and a Guide (see Section 5.4).

At the end, CaiTI generates a report that follows the same format as the therapists’ notes during their treatment sessions. Appendix A reports the details for the 37 dimensions used for day-to-day functioning screening, example questions from CaiTI, and sample responses from the users. Figure 6 shows the smartphone interface for CaiTI’s conversational chatbot, where the various psychotherapeutic interventions applied during different stages of the conversation are annotated.

As CaiTI provides comprehensive daily functioning screening, as presented in Appendix A, some of the dimensions, such as law-abiding, might be sensitive or uncomfortable for users. Therefore, CaiTI offers the option for users to manually select the dimensions to work on with the smartphone interface shown in Figure 7.

CaiTI’s Questioner drives the conversations based on Epsilon-Greedy Q-learning and a GPT-based “Rephraser”. To make the conversation more natural, psychotherapists provide a set of questions they typically ask in each dimension (7 to 11 sample questions). We prompt a GPT-4-based Rephraser (OpenAI, 2023b) to rephrase these questions (structurally instead of semantically) when asking questions.

Each dimension has one related question. The Q-learning agent has 39 states (37 questions, start, and end). We set the learning rate and discount factor to 0.1 and 0.9, respectively. The probability of selecting the best action is set as $\epsilon=0.9$. The therapists determine the initial Q-values for the Q-table based on their empirical evaluation of the “importance” of the dimensions. The Q-value, represents the expected future rewards that can be obtained by taking a given action (next question) in a given state (current question). The Score based on the analysis of user responses is the reward earned in that state.

When the user responds to the question asked by CaiTI, CaiTI first segments the response into individual sentences. For each segmented sentence, CaiTI classifies it into (Dimension, Score), where there are 37 dimensions and 3 scores (Score $\in\{0,1,2\}$) – a total number of 111 classes. In addition, we define 5-class general responses to express Yes, No, Maybe, Question, and Stop (e.g., “Yes”, “I don’t know”, “Stop”, “Maybe”, and “I don’t understand your question”) as well as a mapping table between the Scores and general responses for each dimension. For example, the Score of “Yes” is 0 to the question “Are you showing up for work or school?” in Managing Work/School, while it is 2 to the question “Do you often drink alone?” in Alcohol Abuse.

Since CaiTI asks open-ended questions, user responses are infinitely diverse (YES/NO answers or open-ended responses). With a Score of 2, CaiTI will conduct the psychotherapeutic conversational intervention. Otherwise, CaiTI will ask the next question based on the Q table. When CaiTI meets out-of-context responses, it asks the user to rephrase and follow the process illustrated in Figure  4.

As depicted in Section 4.2 and Figure 4, during the conversation screening process, if the user’s response includes situations that require further attention (with Score = 2), CaiTI will pose a follow-up question to gather more information about the contributing factors to this situation. The follow-up question would start with the simple reflection in MI, a technique where the psychotherapist or counselor mirrors what the client has said. A GPT-4-based ReflectiveSummarizer is prompted to provide the simple reflection, which essentially rephrases or repeats the client’s own words, altering any self-references from the first person to the third person  (OpenAI, 2023b).

The follow-up question initiates a reflection-validation (R-V) process to provide effective empathic validation and support. Given that follow-up responses from the user are open-ended and have infinite possibilities, CaiTI incorporates a R-V Reasoner to determine whether the follow-up response is related to the original response or the question asked in the current state. As illustrated in Scenario 1 in Figure 8, CaiTI will offer empathic validation if the user provides a valid follow-up response. Otherwise, the R-V Guide will assist the user in providing a follow-up response that more accurately describes the situation at hand before proceeding to empathic validation (Scenario 2 in Figure 8).

With a valid follow-up response, a R-V Validator is used to provide empathic validation and support to the user, which incorporates the affective reflection and affirmation techniques in MI. These two techniques aim at demonstrating empathy and understanding of the client’s feelings, creating a supportive environment that validates the client’s experiences and feelings, and fostering a deeper connection.

As described in Section 2.2 and Section 4.2, the CBT process usually includes four steps and CaiTI completes the first step – identifying the situation and issues – for the user based on the historical user responses in the current conversation session. As such, there are three stages remaining: recognizing the negative thoughts (CBT_Stage1), challenging the negative thoughts (CBT_Stage2), and reframing the thoughts and the situations (CBT_Stage3). As the three stages target at different objectives, there are a Questioner, Reasoner, and Guide in each stage.

Figure 9 shows the pipeline for the three-stage CBT process. In each stage, the Questioner poses questions aligned with that stage’s specific objectives. The Reasoner then evaluates the user’s response for its validity and utility. CaiTI progresses to the next stage if the response is deemed valid. If not, the Guide aids the user in crafting a valid response by identifying issues in the current response and suggesting possible improvements for a valid answer. If the user does not provide a valid response after two attempts, CaiTI will conclude the CBT process and recommend that the user seek professional assistance for a more effective and valid CBT experience.

Therapists also point out that an acceptable response involves identifications of cognitive distortion, such as polarized thinking, overgeneralization, emotional reasoning, catastrophizing, and jumping to conclusions. The Reasoner is tasked with recognizing responses containing cognitive distortions as valid, especially for CBT_Stage1 Reasoner. Meanwhile, if the response with cognitive distortions is invalid (e.g., not relevant to the situation), the Guide must take these distortions into account when assisting the user in formulating a valid response.

As illustrated in Section 5.2 to 5.4, overall, GPT-based models have higher performance compared to Llama-based models in achieving the desired functionalities in CaiTI and following the instructions specified in the few-shot prompts, especially in complex tasks, such as: (i) Response Analyzer, which needs to classify the user response into (Dimension, Score) (5 general response classes, 37 dimensions, and 3 scores), and (ii) different Reasonsers, R-V Guide and R-V Validator, where the user responses may span infinitely diverse spectrum and may not fall within the current conversation topic. Llama-based models had a hard time following the instructions in the few-shot prompts when the expressions from the user lacked logical consistency and with cognitive distortions.

Moreover, GPT-based models sometimes add their own interpretation of users’ feelings instead of providing an objective, matter-of-fact output based on the user responses. Llama-based models with few-shot prompts have more stable performance for CBT_Stage2 Guide and CBT_Stage3 Guide, where the user responses are more standard and controlled thanks to the filtering of CBT Reasoners and the tasks, challenging and reframing the negative thoughts, are more straight forward. These observations are expected, as GPT-based models have larger parameters and are pre-trained on a larger corpus of data. Indeed, there are methods that can further improve the quality of the outputs generated by these LLM models, which will be further discussed in Section 9.

20 subjects voluntarily participated (received informed consent from each subject) in our study (approved by the Institutional Review Board), including 10 men and 10 women between 18 and 40 years old from different races. All participants reported having normal hearing and cognition with no history of serious mental or physical illness. All subjects were either students or employed. Each subject was assigned a random subject ID (e.g., S01) for data identification.

Considering the response time and computational resources requirement, the best performing LLM according to the microbenchmarks is implemented for different modules, as illustrated in Figure 9. As discussed in Section 5, different LLMs excel at handling different tasks in CaiTI’s conversation flow. For example, GPT-3.5 turbo is more suitable for providing guide and empathic validation, while GPT-4 is better at reasoning the validity and utility of user response.

As mentioned in Section 1, to accommodate individuals with different needs and with possible memory or vision impairments, particularly the elderly, CaiTI is available in two physical form factors: a customized multi-platform app and a smart speaker (Amazon Alexa). CaiTI integrates custom Alexa Skills with an Amazon Echo device and a Flutter-based application, enabling flexible interaction through voice or text on multiple platforms such as Android, iOS, Windows, and macOS.

Figure 3 depicts the System’s architecture, highlighting the communication pathway where only the conversation text is transmitted between the user interfaces and the server. This design decision is implemented to mitigate the risk of compromising sensitive information that may be inherent in voice data. Voice interactions are facilitated by APIs such as the Alexa Skills Kit (Ale, 2023) and Google’s speech-to-text API (Spe, 2022). The server’s role is to handle all LLM-based tasks, including interpreting text inputs and generating appropriate follow-up questions or psychotherapeutic interventions that are then delivered to the user interface.

Figure 6 shows the smartphone interface for CaiTI’s conversation session, where the various psychotherapeutic interventions applied during different stages of the conversation are annotated. Figure 11 displays the home page of the CaiTI on a smartphone (Figure 11(a)), shows user interactions with CaiTI via voice commands using both the smartphone and an Amazon Echo during in-lab sessions (Figures 11(b) and 11(c)), and depicts a user at home interacting with the CaiTI through text input on a computer (Figure 11(d)).

We first conducted a 14-day study with in-lab and at-home sessions for each subject. The subjects participated in the in-lab session on the first and last days. Afterward, 4 subjects voluntarily participated in a 24-week at-home longitudinal study. A licensed psychotherapist implemented bi-weekly PHQ-9 and GAD-7 assessments with these 4 subjects to measure the effectiveness of CaiTI. Figure 12 shows the modules and tasks implemented in each setup. Subjects were told they were able to unselect some dimensions if they felt uncomfortable through the smartphone interface shown in Figure 7, but they were encouraged to select all. All conversation sessions (user and CaiTI responses, Response Anayzer’s results, Reasoners’ results) are saved for evaluation purposes.

The user responses to the original questions were divided into segments. Each response segment was classified into (Dimension, Score) pairs, with ground truth labeled by the therapists. Dimension classification accuracy (5 classes of general responses and 37 dimensions) reached 97.6%. Figure 13 presents the confusion matrix for the Score of the 7,989 segments with a 99.4% accuracy.

During the 2,013 reflection-validation (R-V) process, R-V Reasoner and R-V Guide were activated 2,120 and 107 times, respectively. As shown in Table 6, CaiTI’s R-V Reasoner almost perfectly identifies all “invalid” follow-up responses from the user with only 1 exception. R-V Reasoner misclassified 42 valid follow-up responses as invalid, which is acceptable in the context of precautionary psychotherapeutic intervention, as it would guide the user to provide valid follow-up responses with better quality. In addition, R-V Guide achieved an accuracy of 96.57%, with only 5 guides being slightly not perfect. The causes for these 5 imperfect guides are overinterpreting the relationship between the follow-up response and the original response and missing some information from the user responses.

The therapists also checked all the 2,013 empathic validations and supports provided by the R-V Validator to the subjects through a majority vote, with 69 being slightly inappropriate. Specifically, when follow-up responses were too brief (under 3 words), CaiTI struggled to comprehend, leading to 24 improper empathic supports. 15 inappropriate instances resulted from inconsistencies between the subjects’ original and follow-up responses. Another 30 inappropriate validations are due to the GPT-based R-V Validator adding its own interpretation of user responses into the empathic validation. Overall, CaiTI effectively delivered empathic validation and support in over 96.5% of instances. Although there are concerns about bias in large language models, the fine-tuned models in this work perform well in user studies and exhibit minimal bias.

Table 7 illustrates the performance of CBT Reasoners and CBT Guides when handling the responses from subjects from all attempts during the three-stage CBT process. It is shown that CBT Reasoners in each stage have high accuracies in identifying the validity and utility of the user responses to meet the psychotherapeutic goal of the CBT process. The CBT Reasoners also achieve high recall, which demonstrates that it is extremely rare for CBT Reasoners to miss any user response that’s not valid or not related.

CBT_Stage1 Guide, CBT_Stage2 Guide, and CBT_Stage3 Guide generated only 3, 3, and 2 less ideal context to guide the subjects. The most common issues for these 8 suboptimal CBT Guides is that they tried to read minds and made excessive assumptions about the relationships of the user responses earlier in the CBT process. None of these suboptimal CBT Guides completely misguide the user or lead them in the wrong direction.

During the 24-week study, we meticulously tracked the number of conversation sessions each subject had per week with CaiTI. As illustrated in Figure 14, the data indicates that all four subjects engaged in dialogue with CaiTI with a frequency of 2 to 3 times daily. Notably, this consistent engagement over the course of the study suggests a sustained use of the system by the participants. The frequency of their interactions with CaiTI did not exhibit a significant decline over time, implying a stable user adaptation and a persistent incorporation of the system into their daily routines. This enduring engagement underlines the utility and user-friendliness of CaiTI, as well as its potential to maintain user interest and interaction over extended periods.

ChatBot Usability Scale (BUS-15) is a recently developed tool to assess end-users’ satisfaction with chatbots (Borsci et al., 2022). Taking BUS-15 into account, we devised the 10 aspects listed in Table  8. Subjects rated the system from 1 (poor) to 5 (excellent) for these 10 aspects on the first and last day of the experiment. In general, subjects gave high ratings to CaiTI. Most subjects thought positively of the conversations with CaiTI and were willing to recommend and continue using CaiTI in the future. Only one subject showed a slightly negative attitude towards CaiTI and gave low scores. This subjective thought that “the technology is too intelligent and makes me worried that AI might be playing a too important role in my daily life” (S19).

A Wilcoxon signed-rank test, a nonparametric statistical within-subject test commonly used in behavioral science (Shin et al., 2022), was performed to compare the subjects’ day-to-day functioning on the first and last days of the 14-day study based on their responses Score on all 37 dimensions (Score $\in\{0,1,2\}$, described in Section 4). The test results show a statistically significant decrease in the dimensions with Score equal to 1 and 2 on the last day, as compared to the first day of the experiment (z = -2.68, p ¡ .01), showing a decrease in dimensions subjects reported with concerns functioning. This suggests an overall reduction in the dimensions that subjects reported as having functional concerns. The results indicate that the subjects’ day-to-day functioning may have improved during the experiment using CaiTI.

In a 24-week longitudinal study, 4 subjects completed additional GAD-7 and PHQ-9 assessments every two weeks. These tools evaluate the severity of anxiety and depression symptoms. With the small sample size (n=4), therapists reviewed the questionnaire results individually. During the study, two subjects improved from moderate-to-severe anxiety and depression to mild levels, initially facing difficulties in functioning but eventually reporting none. One subject progressed from mild to minimal anxiety and depression and reported no functional difficulties. The remaining participant exhibited fluctuating symptoms, ranging from mild to minimal, initially facing some difficulties in functioning, but reporting none by the study’s end. The findings of this study indicate that CaiTI was effective in reducing the severity of anxiety and depression, as well as enhancing daily functioning.