Analyzing sequential activity and travel decisions with interpretable deep inverse reinforcement learning

This study aims to model daily activity-travel decisions of individuals, focusing on what activities people choose throughout the day and when they decide to start and end these activities or travel behaviors. To represent time choices discretely, we divide the day into fixed-length intervals (e.g., 15 minutes). In this way, an individual’s daily activity-travel sequence $Y_{m}$ can be represented as $Y_{m}=\{y_{1},y_{2},...,y_{n}\}$, where $y_{i}$ denotes the activity conducted at time step $i$, and $n$ is the total number of time steps in a day.

Markov decision processes (MDPs) provide a mathematical framework for modeling decision-making processes. An MDP is generally defined as $M=\{S,A,T,R,\gamma\}$, where $S$ is the state space, $A$ is the action space, $T(s,a,s^{\prime})$ is the transition model determining the next state $s^{\prime}$ given the current state $s$ and action $a$, $R$ is the reward function for each state-action pair, indicating the instantaneous reward similar to the utility function in DDCM models, and $\gamma$ is the discount factor. An agent in state $s\in S$ decides the action $a\in A$ based on a policy $\pi$, which specifies a probability distribution over actions. In the context of activity and travel behavior modeling, each MDP component is defined as follows:

• •

  State $s\in S$: The state consists of five travel-related features at the current time step ($Card(S)=5$): (1) Current time step $i\in\{1,2,...,N\}$, (2) Current activity $y_{i}$, (3) Duration of stay at the current activity $d_{i}\in\{1,2...N\}$, (4) Number of activities conducted since the start of the day $p_{i}\in\{1,2...N\}$ (5) Duration since last leaving home $q_{i}$, 0 if currently at home, otherwise $q_{i}\in\{1,2...N\}$.

• •

  Action $a\in A$: An action represents the activity the agent plans to conduct next, which can include home, work, education, shopping, etc. To explicitly account for travel behavior, travel is defined as an additional activity category. Unlike other activities, travel serves as a transition action when people decide to start a new activity. To represent this, we add action masks in our model to restrict the feasible action space based on the current activity. For example, when the user is engaged in an activity, the action space allows for continuing the same activity or transitioning to travel. When the user is currently traveling, the action space includes travel or any other activity.

• •

  Transition Model $T(s,a,s^{\prime})$: For activity-travel behavior analysis, the transition model is deterministic, meaning a given state-action pair will always lead to the same next state. Specifically, at the next time step $i+1$, the activity is determined by the action ($y_{i+1}=a_{i}$). If the agent continues the same activity ($y_{i+1}=y_{i}$), the stay duration at the activity increments by one ($d_{i+1}=d_{i}+1$), and the number of activities since the start of the day remains unchanged ($p_{i+1}=p_{i}$). If the agent transitions to a different activity, the stay duration resets to one ($d_{i+1}=1$), and the number of activities increases by one ($p_{i+1}=p_{i}+1$). The duration since last leaving home increments by one ($q_{i+1}=q_{i}+1$), unless the next activity is home, in which case it resets to zero ($q_{i+1}=0$).

• •

  Reward Function $R(s,a|m)$: For an agent $m$, the reward function $R(s,a|m)$ defines the expected reward of choosing an action $a$ given the current state $s$. To account for individual differences in behavioral preferences, we include socio-demographic features such as income, age, gender, etc., as part of the input to the reward function.

• •

  Policy function $\pi(a|s,m)$: The policy function $\pi(a|s,m)$ describes the probability distribution over actions $a$ that an agent $m$ might take given the current state $s$. While the reward function assigns a value to each state-action pair indicating human preferences, the policy function captures the behavioral patterns of humans.

IRL is a learning paradigm that aims to infer a reward function to explain human preferences and subsequently learn a policy function to replicate human choice behavioral patterns. Like DDCMs, it operates under the assumption that people make decisions to maximize some reward functions, though they are unknown. However, the original IRL is ambiguous and ill-defined, as many optimal policies can explain a set of demonstrations, and many rewards can explain an optimal policy [35]. To address the first ambiguity, [45] developed a probabilistic approach based on the principle of maximum entropy. Building on this, [13] further addressed the second ambiguity with adversarial IRL (AIRL), which introduced a specialized adversarial network to recover disentangled rewards that are invariant to changing dynamics. The original AIRL was proposed for robotic control tasks, and here we adapted it for modeling sequential activity-travel behavior.

AIRL operates within an adversarial framework, comprising a generator and a discriminator. The generator creates realistic trajectories and learns a policy function that maps each state to a probability distribution over actions. The discriminator distinguishes between generated and observed behavior by inferring a reward function, which assigns a value representing the preference of each state-action pair. An overview of the adapted AIRL framework in this study is shown in Figure 2.

The policy network incorporates two types of input features: those related to the current state and those related to the socio-demographic characteristics of an agent. For features of the current state, which are all categorical variables, we learn an embedding matrix for each feature within the model. The matrix maps each category to a latent vector, with the matrix’s width corresponding to the number of possible categories and its length to the dimension of the latent vector. For continuous positive socio-demographic features like income and age, we normalize them to a 0-1 scale before inputting them into the model. For categorical socio-demographic features such as employment type, we use embedding techniques to encode them into a latent space. All the encoded features are then concatenated and fed into a three-layer feedforward network, followed by a Softmax function to output the choice probability of the next activity. As mentioned in Section 3.1.1, we add an action mask before the Softmax function to ensure that only feasible state-action transitions are considered.

The reward network incorporates the current state and individual socio-demographic features as inputs, encoded using the same methods as in the generator. Additionally, to capture the immediate reward of transitioning to the next state, the reward network takes the next state features as inputs, which are encoded similarly to the current state features. To update an unshaped reward invariant to the dynamic environment, the reward function is formulated as follows:

where $g_{\theta}(s_{t},a_{t}|m)$ represents the immediate reward approximator. It captures the intrinsic reward the agent $m$ receives for taking action $a_{t}$ in state $s_{t}$. $h_{\phi}(s_{t}|m)$ and $h_{\phi}(s_{t+1}|m)$ represent the reward shaping components designed to shape the reward function to make it invariant to changing dynamics in the environment. The term $\gamma h_{\phi}(s_{t+1}|m)$ adjusts the reward based on the expected future state $s_{t+1}$, while $h_{\phi}(s_{t}|m)$ adjusts it based on the current state $s_{t}$. This difference helps smooth the reward landscape, making the learned reward function more robust. We use two separate DNNs to approximate $g_{\theta}(*)$ and $h_{\phi}(*)$ separately, each of which is a three-layer feed-forward network based on the encoded state and socio-demographic features.

Based on the reward and policy functions, the discriminator in AIRL is defined as follows:

The discriminator is trained to differentiate between generated and real behavior by minimizing the cross-entropy loss between state-action pairs produced by the generator and those from observed trajectories. The policy function aims to identify an optimal policy that maximizes the expected reward as estimated by the discriminator. To achieve this, we employ a state-of-the-art policy gradient algorithm called Proximal Policy Optimization (PPO) [32] to train the policy network.

The original policy function is represented using DNNs, making it challenging to identify the learned knowledge from individual parameters. To address this issue, we adapt a knowledge distillation method for policy interpretation, initially introduced by [15]. A key insight from their work is that for complex models that learn to discriminate between classes, the knowledge learned is reflected in the choice probabilities assigned to each class. To transfer the knowledge from large, cumbersome models to smaller models, [15] proposed training a smaller model using the class probabilities produced by the large model as “soft targets.” We adapt this idea for interpretation purposes, using a surrogate interpretable model to distill knowledge from the policy network. Specifically, we use a multinomial logit (MNL) model as the surrogate, a common choice for choice analysis that provides comprehensive econometric interpretations [40, 33]. To extract knowledge from the policy network, the surrogate MNL model is trained to maximize the log probability of both ground-truth labels from observed behavior and “soft labels” from the policy network:

where $M$ is the number of users in the training set, $N$ is the number of timesteps. $\bm{y}_{t,m}$ is a one-hot vector denoting the ground-truth activity of user $m$ at time $t$, $\hat{\bm{y}}^{irl}_{t,m}$ is the choice probability vector generated by the pre-trained policy network, and $\hat{\bm{y}}^{mnl}_{t,m}$ is the estimated choice probability of the MNL model. $\alpha\in[0,1]$ is a trade-off weight to balance different losses. When $\alpha=0$, the MNL model is trained completely based on ground-truth labels, while when $\alpha=1$, the model is trained completely based on soft labels generated by the policy network. The MNL model can then be directly interpreted through its learned parameters.

The goal of DIRL is to infer an unknown reward function that explains how people make decisions based on a set of demonstration data. A natural approach for interpreting rewards is to map trajectories in the demonstration data to their corresponding reward values using the pre-trained reward network. By doing so, we establish a direct link between human behavioral patterns and underlying preferences. The reward function assigns a value to each state-action pair, indicating the immediate utility an agent can gain. By applying the pre-trained reward network to each timestep, we can map an activity-travel sequence $Y_{m}$ to a reward sequence $R_{m}=\{r_{1,m},r_{2,m},...r_{N,m}\}$ where $r_{i,m}$ is the inferred reward at the $i$-th timestep for agent $m$. The reward sequence provides a dynamic view of underlying human preferences in sequential behavioral processes. It also enables us to convert discrete behavioral sequences into continuous values, facilitating further analysis.

People can have varying preferences when making sequential activity-travel decisions. For instance, some may have regular, repetitive behavioral patterns, while others explore new and diverse activities with less predictable patterns [27]. To identify different types of decision-makers, we use a clustering technique to group individuals based on their corresponding reward sequences. People within the same cluster share similar preferences for activity-travel planning. In this case, without loss of generalizability, we employ the k-means algorithm [16], a popular method for partitioning data into $k$ distinct clusters.

The reward values estimated from the reward function represent the immediate utility an individual can gain. In sequential decision-making, it is important to consider not only the immediate reward but also the long-term expected utility. The reward function allows us to quantify the long-term expected return $U^{(i)}_{m}$ for an agent $m$ starting from timestep $i$ as follows:

In our case, without loss of generality, $i$ is set to 1, representing the accumulated return starting from midnight. This long-term expected utility can compress the reward sequence into a single value, representing the overall utility an individual gains from the entire sequential planning process. Using this quantified long-term utility, we can compare the “values” of different activity-travel patterns. This comparison enables us to understand which decision-making process is optimal, at least from the perspective of the DIRL model.

Table 2 presents the performance of the MNL model trained with and without soft labels generated by the policy network. The results indicate that incorporating soft labels in the loss function enhances the MNL’s performance across all evaluation metrics. This is because compared to using ground-truth activities as labels, the soft labels generated by the policy network provide richer information about the relative choice probabilities among different activities. Surprisingly, when $\alpha$ is set to 1, meaning the MNL is trained entirely based on the soft labels predicted by the policy network, it achieves the highest performance in terms of accuracy and edit distance, and similarly strong performance in BLEU score compared to other $\alpha$ settings. This suggests that even without the ground-truth activities, the choice probabilities learned by the policy network can effectively capture real activity-travel behaviors.

This section analyzes the behavioral patterns derived from the policy network by examining the coefficients of the surrogate MNL model. To ensure the robustness of our interpretations, we employ a bootstrapping strategy. Specifically, we randomly sample 80% of the observations with replacement to train the surrogate MNL model, repeating this process 100 times. Table 3 presents the average coefficients and the [0.025, 0.975] confidence intervals for each input variable based on the 100 bootstrap samples. Variables with inconsistent directional impacts within these intervals are highlighted in grey. The results show that most variables maintain consistent directional impacts over 95% of the time, validating the robustness of our interpretations. Features with consistent directional impacts in most experiments are likely to significantly influence human activity and travel behavior patterns, while those with inconsistent impacts may be less significant in explaining these patterns.

The current activity an agent is engaged in significantly influences their next activity choice. When the next activity is home, work, or education, there are positive coefficients for the same activity in the current time step. This means individuals tend to continue with the same activity in the next time step. Conversely, when the current activity falls under the category of others, choosing travel in the next time step is associated with positive coefficients, indicating that individuals are more likely to transition to travel. This pattern is reasonable, as the duration spent at home, work, and education is typically longer, whereas other activities generally have shorter durations, as illustrated in Figure 3.

By examining the time slots with positive coefficients, we can identify the most common periods for different activities. Home activities are most common early in the morning (0-6h) and in the afternoon and evening (14-24h). The most common work hours are between 6-16h, while school activities peak from 6-12h. For other activities, all time intervals show negative coefficients, likely because these activities occupy a smaller portion of individuals’ daily lives. Travel is more frequent during 6-8h and 16-20h, corresponding to the morning and evening peak hours for commuting.

The duration of staying at the current activity positively correlates with choosing travel as the next activity. This is reasonable as people are more likely to move on to another activity after spending a significant amount of time in one place. The number of activities since the start of the day shows positive coefficients for home and negative coefficients for other activities. This indicates that as the number of activities increases, people are more likely to return home and less likely to engage in out-of-home activities such as work, school, and others. Additionally, the duration of out-of-home activities is positively correlated with both travel and home, while negatively correlated with work, school, and other activities. This further supports the idea that as out-of-home duration increases, people tend to travel back home.

Behavioral patterns vary across different age groups. Increased age is positively correlated with work and other activities, indicating that older individuals are more likely to engage in these activities. This is reasonable, as older people often have more social and family responsibilities. Gender differences also emerge in activity-travel planning. Females are positively correlated with staying at home and engaging in other activities, while negatively correlated with work, school, and travel. This suggests that females are more likely to stay at home or participate in other activities and less likely to go to work or school, possibly due to a greater share of household duties. Income level also influences activity choices. High-income individuals are more likely to go to work and less likely to go to school, which is expected as schoolgoers typically do not have an income.

Different employment types clearly correspond to distinct activity planning patterns. Homemakers are more likely to stay at home or engage in other activities rather than work, school, or travel. This is reasonable as they often take more household responsibilities, such as shopping and picking up children. Full-time students tend to go to school, while full-time, part-time, and self-employed individuals are more likely to go to work. Notably, the coefficient for travel among full-time employees is higher than those for part-time and self-employed individuals, indicating that full-time employees are more likely to travel compared to part-time and self-employed individuals. This difference suggests that work modes influence travel tendencies: full-time employees have fixed schedules, resulting in regular commuting trips, whereas part-time and self-employed individuals have more flexibility in their time and location choices. Retired and unemployed individuals, as well as domestic workers, show positive coefficients for staying at home and negative coefficients for work, school, or travel. This indicates that these individuals tend to stay at home, which is reasonable since they do not need to go to office or school and have fewer travel requirements.

Figure 5 provides an intuitive illustration of the learned reward values at different steps in example activity-travel trajectories. Notably, the steps where individuals transition from an activity to travel are typically associated with the lowest instant rewards in the trajectory. This aligns with the intuition that most people do not enjoy traveling and only do so to fulfill necessary activities. Another interesting observation is that the instant reward for the same activity can change as the stay duration increases. Generally, the reward first increases but starts to decrease once the stay duration exceeds a certain threshold, indicating that people have a preferred stay duration for different activities.

Figure 6 depicts the distribution of reward values across various activity types. The mean reward values for home, work, and education activities are higher compared to other activities. This is likely because home, work, and education are regular, expected parts of daily life, whereas activities like shopping or medical visits arise from irregular personal or household needs that are less predictable. On the other hand, travel has the lowest average reward, which aligns with previous findings that it is generally perceived as a necessary but often unpleasant activity. The variance in reward values is also noteworthy. The distribution of rewards for other activities is more dispersed compared to home, work, and education. This is reasonable since “other” activities encompass a wide range of possible categories, such as shopping, pick-up drop-off, and medical visits, each with different reward values. The reward values for travel show two peaks. This can be explained by the extremely low rewards associated with the initial transition into travel, while subsequent travel steps are associated with higher reward values, as illustrated in Figure 5.

Reward values at different timesteps are correlated with each other. To understand these sequence-level patterns, we use a k-means clustering technique, as described in Section 3.2.2, to group reward sequences into different types. The cluster size is set to 4, which is the optimal group size determined using the elbow method. Figure 7 shows the clustering results of different reward sequences and their corresponding activity patterns. Cluster 1 and 2 correspond to users with regular travel patterns at similar times while cluster 3 and 4 correspond to users with less regular and predictable travel patterns. This in some way matches with the well-established explorer and returner theory [27], which distinguishes travelers as either having regular, repetitive movements or exploring new and diverse activities with less predictable movement patterns.

To understand the individual variance between different groups, we analyze the employment type composition as shown in Figure 8. Interestingly, each group is dominated by different employment types. Cluster 1 is primarily composed of full-time students, exhibiting representative school patterns. Cluster 2 is dominated by full-time employees, which aligns with typical commuting patterns. Cluster 3 includes a mix of full-time students and full-time employees, corresponding to individuals with regular work or school schedules who also engage in additional activities in the afternoon or evening. Cluster 4 is dominated by homemakers and retired individuals, explaining why they perform other activities during the morning hours. This demonstrates the capability of reward values to reflect and differentiate between various activity patterns and individual behaviors.

Based on the reward values at each step, we can estimate the long-term return of the activity sequence in a day, which compresses the impact of different activities at different time steps into a single value. The long-term return in a day can in some way indicate the overall utility an individual can get from his or her daily activity planning. By sorting the activity sequence based on their corresponding return value, we can then have a sense of what types of activity planning are more optimal than others, at least based on the understanding of the DIRL model. Figure 9 shows 500 activity sequences sorted by their associated return values. Activity sequences on the left side are associated with higher returns. The results clearly demonstrate a change from regular to irregular activity patterns, indicating that the model assigns higher return values for individuals with regular activity patterns. Activity sequences with top return values are school patterns, followed by regular commuting patterns. This corresponds to the results in Figure 6 that the DIRL model assigns a higher reward value to education than to work. This can be because people have a better sense of accompliment and get better long-term investiment when experiencing education.

To explain the individual variance in return values associated with daily activity sequences, we divided trajectories into four groups based on return quantiles. Figure 10 shows the socio-demographic features of these four groups. Interestingly, the employment type compositions associated with user groups having different return levels closely align with the types of activity planners identified through the clustering technique discussed in the previous section. The top return group consists mostly of students. The mid-high return group is a mix of full-time employees and students. The mid-low return group is mainly full-time employees, while the low return group is predominantly homemakers. This distribution indicates that return values can reflect differences in employment types among individuals. Regarding age, groups with lower returns tend to include older individuals. This is likely because students, who receive higher returns, are typically younger, while homemakers and retired individuals, who receive lower returns, are generally older. There are also gender differences in return values: in the top return group, only 20% of individuals are female, whereas in the bottom return group, more than 70% are female. This suggests that females are typically associated with lower daily returns, possibly due to taking on more home duties that do not directly generate socially recognized value. The relationship between income and return groups forms a “$\cap$” shape: the top return group has the lowest average income, which is reasonable as it mainly consists of full-time students with no income. Conversely, from the mid-high to low return groups, as the average income decreases, the return on their daily activities also decreases. This indicates that the return values derived from activity sequences have the potential to reflect individual income levels.