Persuading to Prepare for Quitting Smoking with a Virtual Coach: Using States and User Characteristics to Predict Behavior

Knowing the state a persuadee is in may help to predict their behavior after persuading them with different persuasive strategies (i.e., actions). The behavior in our case is the effort people spend on their preparatory activities, which is captured by our reward function. We compared two approaches for predicting the reward: 1) the mean reward per action, and 2) the mean reward per action and state. We used leave-one-out cross-validation for the 671 participants with at least one transition sample to compare the two approaches based on the mean $L_{1}$-error and its Bayesian 95% credible interval (CI) Oliphant (2006) per state. In contrast to classical confidence intervals, Bayesian CIs provide information on the most likely values (i.e., a likely range) Hoekstra et al. (2014). We regard non-overlapping 95% CIs as a credible indication that values are different, both for this research question and the subsequent ones.

Considering the state tends to result in lower $L_{1}$-errors for predicting the reward than not considering the state (Figure 1). This makes sense, as the mean reward strongly differs between states. For example, while state 000 has a mean reward of -0.52, state 111 has one of 0.25 (see the red line in Figure 1). In such states with mean rewards much lower or higher than the overall mean reward, the advantage of considering states for the reward prediction is pronounced with the 95% CIs for the two approaches not overlapping. This provides a credible indication that considering states performs better. For states with mean rewards more similar to the overall mean reward, on the other hand, the 95% CIs for the two approaches tend to overlap. So there is no credible indication that one of the two approaches is better for those states.

Considering the state a persuadee is in helps to predict the effort they spend on an activity, as long as the state is one in which people tend to spend much less or more effort on activities than on average. Using features derived from the COM-B model, we obtained such states.

Q2: How well can states derived from the COM-B model predict states after persuasive attempts?

Ideally, we want to persuade a person in such a way that they move to a state in which they are likely to again be persuaded to spend a lot of effort on an activity. Therefore, we need to be able to predict the state after a persuasive attempt. Using leave-one-out cross-validation, we compared three approaches for predicting the next states for the samples from the left-out person: 1) assigning an equal probability to all states, 2) predicting that people stay in their current state, and 3) using the transition function estimated from the training data. We compared the three approaches based on the mean likelihood of the next state and its 95% CI per state. A higher likelihood suggests that next states can be predicted better.

Figure 2 shows that considering the current state, by either predicting that people stay in their current state or assigning a probability to next states based on the estimated transition function, leads to a higher mean likelihood of next states than assigning an equal probability to all next states. This shows that state transitions do not occur uniformly at random. Notably, predicting that people stay in their current state leads to the highest mean likelihood of next states in three of the eight states. These states are states 000, 010, and 111. In each of these states, the mean for predicting that people stay in their current state is highest and the corresponding 95% CI does not overlap with the ones for the other two approaches. This shows the high probability of staying in those three states, which are states with either very low or very high mean rewards (Figure 1).

Our results show that considering the current state a persuadee is in helps to predict their next state after a persuasive attempt. For persuadees who are in states in which people tend to spend very little or very much effort on their activities, this next state tends to be the same as the current one. This means that if we just persuade people as we did in the study used to collect data, we will have limited success in moving people from low-effort to higher-effort states. Though once people are in higher-effort states, they are likely to stay.

Q3: What is the effect of (multiple) optimal persuasive attempts on persuadees’ states?

We would like that people ultimately move to the states in which they are most likely to be persuaded to spend a lot of effort on activities. Starting from an equal distribution of people across the states, we calculated the percentage of people in each state after following the optimal policy $\pi^{*}$ for a certain number of time steps. $\pi^{*}$ was computed via value iteration based on all gathered samples. Table 7 in the Appendix shows $\pi^{*}$.

Figure 3 depicts the transition function under $\pi^{*}$. It is evident that people tend to move to better states or stay in the best state (blue lines). With better states we mean states with higher $V^{*}$. In fact, for each state, there is a probability of at least $\frac{1}{|S|}$ that a person moves to a better state. And once people have reached the best state, which is state 111, there is a high probability of 0.8 that they stay there. However, there are some red lines in Figure 3 as well. These lines show that people sometimes move to worse states or stay in the worst state after being persuaded based on $\pi^{*}$. This happens especially for states with lower $V^{*}$ such as states 000 and 010. For both of these states, there is also a relatively high probability of staying in them. For example, there is a probability of 0.41 that people stay in state 000 once there. Yet, people can also move from states with relatively high $V^{*}$ to states with low $V^{*}$. For state 011, for instance, there is a probability of 0.22 that people move to the lower-value state 010.

Besides the short-term effects of following $\pi^{*}$, we are also interested in the long-term effects when using multiple persuasive attempts. The results of simulating transitions for applying $\pi^{*}$ for up to 20 time steps are shown in Figure 4. It is evident that compared to the initial state distribution with an equal number of people in each state, more people are in state 111 and fewer people in all other states after 20 time steps. Given that state 111 is the state with the highest value, people thus tend to move to the best state. In fact, 62.61% of people and thus more than half are in state 111 after 20 time steps. However, there are always some people in the states with lower values. For example, 6.63% of people are in state 000, the state with the lowest value, after 20 time steps.

While persuading people optimally multiple times allows most of them to move to and stay in the state in which they are expected to spend the most effort on their activities, a few people remain in the state in which they are expected to spend the least effort on their activities.

Q4: How do optimal and sub-optimal persuasive attempts compare in their effect on behavior?

Once we are able to predict states, we would like to choose effective sequences of persuasive strategies. Yet, it is not clear how much the choice of persuasive strategy matters when it comes to the effort people spend on their activities over time. We calculated the mean reward per transition over time when following 1) the optimal policy $\pi^{*}$, 2) the worst policy $\pi^{-}$, and 3) the average policy $\pi^{\sim}$. $\pi^{\sim}$ is a theoretical policy for comparison purposes in which each action is taken $\frac{1}{|A|}$ times for each person at each time step, where $|A|$ is the number of actions. We considered two initial state distributions, namely, the distributions across states in the first session of our study based on a) all people and b) only those people whose first reward was in the lowest 25%-percentile of all first rewards. Distributions are from our study’s first session to represent a general population of people who have never been persuaded to do preparatory activities. We further specifically look at people who initially spend very little effort on their activities when persuaded randomly as at the start of our study, because it is more beneficial to coach people who are not yet performing well.

The mean reward for $\pi^{*}$ is highest for all time steps and increases over time for an initial state distribution that is based on all people (Figure 5). After 100 time steps, the mean reward per transition is 0.17 and therewith above the 50%-percentile of rewards for the first session of 0.13. This means that the mean reward is increased compared to the actual mean reward we observed in session 1. In contrast, the mean reward drops for the other two policies and is only 0.02 for $\pi^{\sim}$ and -0.13 for $\pi^{-}$ after 100 time steps. The former falls in the 40–50%-percentile of rewards for the first session and the latter in the 30–40%-percentile. Hence, the difference in mean reward between the three policies increases over time. We also observe this if we consider the initial state distribution of only those people with low rewards for the first session. For example, the difference between $\pi^{*}$ and $\pi^{\sim}$ increases from 0.08 to 0.15 and thus almost doubles.

These findings show that it matters, both for people overall and for people who are not performing well initially, how we persuade them to do preparatory activities for quitting smoking. Choosing how to persuade people based on an optimal RL-policy thereby performs better than doing so based on a worst or an average RL-policy.

Q5: How does predicting behavior based on user characteristics compare to doing so based on states?

An alternative to using states to predict behavior is using user characteristics. This alternative has the advantage that data on such characteristics do not need to be collected before each persuasive attempt. To compare the use of user characteristics to the one of states, we selected three user characteristics in a similar fashion as the three state features. More precisely, we first turned the user characteristics into binary variables based on whether their value was greater than or equal to the mean (1) or less than the mean (0). Then we iteratively selected the variable with the largest difference in reward when the variable is 0 compared to when it is 1. This is because when the reward is very similar for both values of a variable, it does not improve the reward prediction very much to consider the value of the variable. We considered two different sets of candidate variables. First, we considered only the pre-characteristics and thus data that we can collect from people without having to provide any information about the activities (i.e., we excluded people’s involvement in their activities). Second, we considered all characteristics (i.e., we also included involvement). The selected characteristics in the first case were the Transtheoretical Model (TTM)-stage for becoming physically active, conscientiousness, and smoking status; the ones in the second case were involvement, physical activity identity, and smoking status. For each case, we created a user characteristic state space of size $2^{3}=8$ analogously to the case of state features. Based on these state spaces, we computed the mean $L_{1}$-error for predicting the reward using leave-one-out cross-validation. Our baselines were predicting the reward based on 1) the overall mean reward, 2) the mean reward per action, 3) and the mean reward per action and state.

Figure 6 shows that predicting rewards based on user characteristics in addition to actions outperforms predicting the overall mean reward. Of the two ways of predicting rewards based on user characteristics, the one that includes people’s involvement in their assigned activities leads to a lower $L_{1}$-error. More precisely, the mean $L_{1}$-error is 0.43 for user characteristics with involvement, and 0.45 when excluding involvement. The two 95% CIs thereby do not overlap, providing a credible indication that the mean $L_{1}$-error is lower for the former than for the latter. However, none of the two ways of predicting rewards based on user characteristics performs better than using states, with the latter leading to a mean $L_{1}$-error of 0.41. While the 95% CI for predicting rewards based on states overlaps with the one for predicting rewards based on user characteristics including involvement, it does not overlap with the one for using only user pre-characteristics.

These results provide a credible indication that using states allows us to better predict the effort people spend on their activities than using only user characteristics that we can collect data on without having to tell people about the activities. If we include people’s involvement in the activities as a user characteristic, however, there is no longer a credible indication that using states outperforms using user characteristics.

Q6: How does incorporating users’ similarity based on characteristics, besides the consideration of states, improve the prediction of behavior?

While user characteristics alone may not help to predict behavior compared to states, they may do so in combination with states: people with different characteristics may respond differently to a persuasive attempt in a certain state. We thus examine the effect of incorporating people’s similarity, based on user characteristics, on our ability to predict the effort people spend on their activities. We do so by weighting observed samples differently for each persuadee, whereby a larger weight is given to samples from people more similar to the persuadee. Using different user characteristics and weighting parameters, we tried a total of 68 configurations for weighting samples based on similarity (see Appendix). We here report the results for the configuration with the lowest mean $L_{1}$-error based on leave-one-out cross-validation. This best configuration used people’s involvement in their activities to measure similarity.

Even though the mean $L_{1}$-error is lower for incorporating users’ similarity than for the original approach without similarity, the 95% CIs overlap (see the two rightmost bars in Figure 6).

Incorporating users’ similarity besides the consideration of states appears to offer some improvement, but there is no credible indication that it allows us to better predict the effort users spend on preparatory activities after persuasive attempts.

The main limitation of our work is the data it is based on. While we did gather data from human subjects, we did not assess the effects of our approaches on the actual behavior or states of these humans. Instead, we performed leave-one-out cross-validation and simulations. The primary reason is that this allowed us to test a large number of approaches while staying within a reasonable budget. The best-performing approaches can then be tested in the wild in the future. When doing so, however, several additional factors may need to be addressed. This is because all of our approaches assume the transitions between states and the effort people spend on their activities to be stationary. Stationary here means that the transition probabilities and the mean effort people spend for combinations of states and actions do not change. But intuitively, such changes may occur. For instance, repeatedly sending the same persuasive strategy may make it less effective Thomas et al. (2017), but could also help to strengthen the link between cue and response for action planning Schwerdtfeger et al. (2012) or to scrutinize arguments objectively Cacioppo and Petty (1985). One approach to address the effects of such repetitions is the work by Mintz et al. (2020) on non-stationary bandits. Moreover, once people move beyond preparatory activities and start to actually change their behavior, habits may form after several weeks Gardner and Rebar (2019). Such habits may reduce the cognitive effort and awareness required to do a behavior Gardner and Rebar (2019). One could address this by including information on habits in the state description (e.g., Zhang et al. (2022)).

A more general limitation of our work is the way we defined our problem. First, our state description is based on the COM-B self-evaluation questionnaire and only a subset of the questions therein. While this is a good starting point as our results show, other features, potentially derived from other theories, could be useful. For example, physical capability may play a role when people are to be persuaded to do more complex tasks such as going for a run. Importantly, however, not all people may be willing to answer many questions in each session. So it may be beneficial to either limit the number of questions for all people, or to give people the option to answer additional questions for more precise tailoring (e.g., Hors-Fraile et al. (2019)). Second, our results are based on five widely used persuasive strategies that we deemed to be applicable in our context. Given the large number of other strategies, it is possible that user characteristics play a more important role in explaining the effectiveness of other strategies. Notably, however, there is also ample literature suggesting the importance of user characteristics for the persuasive strategies we used (e.g., Oyibo and Vassileva (2017); Thomas et al. (2017); Zalake et al. (2021)). Third, we measured people’s response to persuasive attempts based on the self-reported effort they spent on their activities. It would be interesting to see whether our findings also hold when a more objective measure of behavior is used. Lastly, another interesting direction to improve our model is to use Bayesian RL. This allows one to incorporate prior information about the dynamics in a flexible manner as well as to consider the uncertainty in the learned parameters when making decisions Ross and Pineau (2008); Ghavamzadeh et al. (2015). For example, one can model relations between state features using a dynamic Bayesian network Ross and Pineau (2008). This may be useful, as behavior models such as COM-B specify relations between predictors of behavior.

We want to make informed decisions on which components to use in persuasion algorithms for eHealth applications for behavior change that are effective as well as more cost-effective and user-friendly by reducing the amount of required human data. Therefore, a better understanding of the components’ individual effects on predicting behavior after persuasive attempts is welcome. We have thus compared the use of states and user characteristics, and a combination thereof, in predicting behavior after persuasive attempts in the context of preparing for quitting smoking with a virtual coach. Our results lend support to the idea of considering states and the user characteristic “involvement” in persuasion algorithms for behavior change. Research on smoking cessation can directly build on these insights and examine the use of these components in a full application. Moreover, both components seem to be domain-independent measures that could also be used in eHealth applications for other behaviors.

This work is part of the multidisciplinary research project Perfect Fit, which is supported by several funders organized by the Netherlands Organization for Scientific Research (NWO), program Commit2Data - Big Data & Health (project number 628.011.211). Besides NWO, the funders include the Netherlands Organisation for Health Research and Development (ZonMw), Hartstichting, the Ministry of Health, Welfare and Sport (VWS), Health Holland, and the Netherlands eScience Center. The authors acknowledge the help they received from Eline Meijer in formulating the preparatory activities and from Mitchell Kesteloo in hosting the virtual coach on a server. The authors further thank the three anonymous reviewers for their helpful suggestions.