Alexa Depression and Anxiety Self-tests: A Preliminary Analysis of User Experience and Trust

Mental health problems are a growing global challenge affecting people of all different backgrounds, ages, and socioeconomic status (Woodward et al., 2019). To tackle this challenge, mental health resources are increasingly available online and via mobile apps (Ly et al., 2017; Morris et al., 2015; Kamita et al., 2019). The proliferation of conversational agents has made them attractive for health applications (Laranjo et al., 2018) and mental health (Vaidyam et al., 2019). Notable chatbots that monitor mood and use aspects of CBT to help users deal with anxiety and depression include Woebot (Stiles-Shields, [n.d.]) and TESS (Fulmer et al., 2018).

The advancements and rapid adoption of spoken conversational agents, such as Siri and Alexa, make them attractive as a channel for providing mental health resources to users (Maharjan et al., 2019). Spoken conversational agents interact with users via spoken natural language. For mental health applications, this requires users to vocalize responses about their mental health status, which is different than typing responses to a chatbot or on a website. One study explained that some people are more likely to have truthful interactions about their mental health with technology than wth mental health professionals (Ravichander and Black, [n.d.]).

This paper presents work-in-progress findings of an Alexa Skill we developed that performs depression and anxiety self-tests. Current Alexa Skills focus on guiding, educating, and helping users manage mental health issues. Some Alexa Skills examples include management for anxiety and stress through advice sessions (Anti Anxiety, Anxiety Stress), assisting people with depression by providing tasks to boost their mood (Mental Health Day Manager), management advice and education for children and teenagers dealing anger, stress, anxiety, and depression (Mental Health Spies), and targeted exercises depending on the situation (work, studies, life) causing the user stress (Mindscape). One study used Alexa to monitor a user’s mental health behaviors and symptoms, requiring users to self-report data on sleep, mood, and activity levels (Maharjan et al., 2019). However, further studies are required to provide evidence regarding the efficacy of delivering mental health resources via spoken conversational agents.

Our contributions are: (1) We developed an Alexa Skill that allows users to conduct depression and anxiety self-tests and makes exercise recommendations to alleviate anxiety and depression symptoms; (2) We conducted a pilot study with 10 participants to assess the usability of our Alexa Skill.

The average age of the participants was 21.6 years old and 8/10 were male. 9/10 of the participants used conversational agents. 7/10 of the participants also indicated that they had not previously used an app for mental well-being. After the two-week period, 7/10 participants indicated that they rarely used our Alexa Skill.

Figure 1 shows the anxiety and depression scores completed using a webpage (online) vs using Alexa. Given that the majority of the participants rarely used the Alexa Skill, we cannot attribute the change in anxiety score or depression scores to the use of the Alexa Skill or to the delivery of the self-tests via Alexa.

Figure 3 shows the user experience scores of the Alexa Skill. The participants found the app’s attractiveness, dependability, and stimulation to be above average, but attractiveness, perspicuity, efficiency, and novelty were scored below average. Figure 3 shows the trust scores for the Alexa Skill. We conclude, participants mostly trusted the app.

This pilot showed a willingness from participants to trust Alexa with personal information such as depression and anxiety scores. The user experience scores of the Alexa Skill also showed that participants considered it lacking in efficiency and novelty.

The design of this study, the implementation of the Alexa Skill, and conducting the pilot, highlighted a number of challenges when it comes to the development of an Alexa Skill for the mental health space.

Cold Start Problem: The first user interactions with an Alexa Skills with multi-step or branching dialogues can be challenging for new users. New users do not know the possibility of dialogue flows, the intent recognitions that have been programmed into the Alexa Skill, and the responses that are valid (how to respond to a question from the Alexa Skill). While Alexa Skills can be deployed with a user manual or by walking the user through a tutorial, complex skills may require regular use and various trials before the user is comfortable interacting with the Alexa Skill. In our pre-pilot tests, users struggled when interacting with our Alexa Skill because they gave responses that were not captured by the intent recognition rules we had programmed. Careful design must be planned to ensure intuitive interactions between the user and the Alexa Skill, as any difficulties are bound to discourage users from future use. This is especially important when designing spoken conversational agent apps for mental health, where users may be experiencing distress when they decide to use the Skill.

Robust Dialogues: Alexa Skills with deep dialogues need robust handling of user responses. In our Alexa Skill, the depression and the anxiety self-tests involved Alexa reading multiple questions and waiting for a user response to each question. During testing, some users were frustrated by having to repeat the depression or the anxiety self-test because the speech recognition of Alexa did not understand their response, or because the Skill closed due to an Alexa error or an error in our Skill. If a user is experiencing distress, this type of experience may worsen the user’s mental state or fail to provide the support intended by the app/Skill.

Handling expression of emotions: Our Alexa Skill allowed users to state how they were feeling using single words. This aspect of our Alexa Skill was brittle, since expressing a word not included in our list of emotions meant that Alexa would ask the user to repeat themselves. Ideally, our long-term goal is to support all expressions of emotions, with Alexa being able to acknowledge what the user is experiencing. We believe this acknowledgment can help users in isolation and experiencing distress, but it poses technical challenges in natural language understanding. Some of the users that tested our Skill also expressed their emotions by referring to physical symptoms associated with an emotion they were feeling, i.e. ”sweaty palms”, ”heart racing”, which poses the additional challenge of knowing the association between a body response and an emotion.

Engagement with the self-test Alexa Skill: Engagement with health support technologies is concerning. Related literature in mobile health apps for mental health reports users’ uptake and engagement challenges (Ng et al., [n.d.]). In our pilot, we observed similar trends where participants rarely used the Alexa Skill. Enhancing the engagement of conversational agents through storytelling (Battaglino and Bickmore, [n.d.]), personalization (Kocaballi et al., 2019), and affect (Callejas et al., 2011) has been proposed in the past. However, further research is needed to understand the design implication of such features for spoken conversational agents in the mental health context.