Towards Explainable Goal Recognition Using Weight of Evidence (WoE): A Human-Centered Approach

We defined the observer’s worldview within the framework of Goal MDPs as follows:

• •

  State Space $S$: Represents the snapshot of the world, including the player’s position, walls, boxes, and storage locations at any given point in time.

• •

  Action Space $A$: Encompasses the various actions the player can take, either moving or pushing the box(es) in one of the following directions: up, down, right, or left.

• •

  Cost Function $C(s,a,s^{\prime})$: Assigns a cost for each action to encourage making the fewest amount of moves.

• •

  Goal State $S_{G}$: Includes all possible goals that the player can achieve within the current state of the world. Each goal represents a desired configuration of boxes in relation to storage locations.

Table  1 presents different goal representations based on the original game version. In versions 1 and 2, the player can push one box at a time, whereas in version 3, they can push two boxes at a time to goal locations. The goal recognition problems include a number of competing goal hypotheses: multiple possible plans to achieve a goal, two sequential goals with interleaved plans to achieve them, or failed/unsolvable plans (See Appendix A for screenshots of different scenarios).

We additionally varied scenarios to present different rationality levels, including rational (optimal or suboptimal) behaviors and irrational behaviors. For optimal behavior, we assumed that observers would form a simple notion of optimal behavior, where the player takes the shortest path toward a goal. In the suboptimal behavior scenario, the player either chooses a longer path toward a goal (suboptimal plan) or deviates from a rational action in an observed sequence of a particular goal plan (e.g., the agent’s goal may have changed). We also included irrational behaviors where the player fails to complete the task (e.g., getting stuck in a dead-end state) to observe how these actions would be interpreted by participants.

The participant’s task was divided into the following phases:

1. 1.

   Watch an instructional video to introduce the task and game rules.

2. 2.

   For each of the 18 different scenarios (three games, six scenarios per game):

   • •

     Watch a video clip in which a player tries to achieve the task (Figure 2).

   • •

     After watching the observed actions sequence (plan’s completion percentage $\textit{min}=0.25\%,\quad\textit{median}=0.53\%,\quad\textit{max}=0.83\%$), predict which goal location the player is trying to get to, and, accordingly, assign a likelihood (with one as the least likely and five as the most likely) of each goal. This prediction task is not central to the objectives of this study but was used to engage participants in reasoning about behavior.

   • •

     Provide reasons for your prediction. Participants were required to answer specific questions based on the condition they were in.

Each participant was randomly assigned to one of three conditions:

• •

  ‘Why’ condition: participants were asked to: ”Explain why you have rated that/those goal(s) as the most likely?”

• •

  ‘Why-not’ condition: participants were asked to: ”Explain why you have not rated that/those goal(s) as the most likely?”

• •

  ‘Dual’ condition: participants were asked to explain both why and why you have not in that order.

We collected data for the first and second conditions to analyze the differences between why and why not, and for the third condition to analyze how people answer why not if they have already answered why, and how the answer of why differs if they know there is a why not.

We recruited 36 participants (22 male, 14 female), allocated evenly and randomly to each condition, aged between 20 and 65, with a mean age of 38. We limited the study to participants from the United States who are fluent in English. Recruitment was conducted via Amazon Mechanical Turk. Participants were compensated $6.50 for completing the task and a bonus of $3.50 for providing more thoughtful answers.

With three different game versions, six scenarios per game, and 12 participants per condition, a total of 864 textual data points were collected (Table 2). We used several methods to filter out deceptive participants. We excluded explanations with fewer than three words or containing gibberish. We also used the time taken to complete the survey as a threshold. This left us with a total of 828 explanations.

We used participants’ open-ended explanations to better identify the concepts they used to explain the player’s predicted goal. The word count of given answers within the dataset is between 1 and 98 words ($\bar{x}_{1}=22.96$, $\sigma_{1}=15.52$) for the first condition, 3 and 81 words ($\bar{x}_{2}=26.57$, $\sigma_{2}=15.63$) for the second condition, and 1 and 64 words ($\bar{x}_{3}=20.38$, $\sigma_{3}=11.65$) for the third condition.

We defined the worldview of shoutcasters, i.e. observers, within the framework of Goal MDPs as follows:

• •

  State Space $S$: Encapsulates the game environment’s configurations and conditions at any given time. This includes the positions of units, resources, and other relevant game elements.

• •

  Action Space $A$: Includes low-level actions of specific game units and high-level actions related to game strategies and tactics (sequences of actions).

• •

  Cost Function $C(s,a,s^{\prime})$: Quantifies the value of states and actions in terms of progress toward winning.

• •

  Goal State $S_{G}$: Comprises both subgoals (intermediate objectives that players or teams aim to achieve) and the main goal (the ultimate objective, typically winning the game).

We analyzed a dataset comprising professional StarCraft tournament videos (?), where expert shoutcasters provided commentary. We identified key instances where shoutcasters made predictions about players’ goals and strategies and coded these instances to capture the underlying concepts used in their explanations. We clustered the dataset to ensure representative sampling.

We obtained the dataset from ? (?), which was collected from professional StarCraft tournaments available as videos on demand from 2016 and 2017. They selected 10 matches and then randomly chose one game from each match. Each of the 10 videos features two shoutcasters (expert commentators) providing commentary.

To obtain representative samples, we identified six clusters within the dataset (1387 instances divided by 6 clusters equals approximately 231 instances per cluster). We then randomly selected one sample of 50 instances from each cluster. The resulting sample size was 300 instances (6 clusters * 50 instances each). We only considered instances involving predictions, specifically when shoutcasters explain their recognition process of what the agents/players aim to achieve (their goals) and so ended up having a total of 132 instances out of the six samples. As the data source is public, we provided supplementary material of coded data to support future research.

We developed the conceptual model by integrating insights from both studies. Initially, we built a model based on the findings from the Sokoban study. This model was then extended to encompass the complexities of the Starcraft study, ensuring that it accurately reflects both studies’ dynamics and unique aspects. We present our conceptual model of an explainable Goal Recognition (GR) agent in Figure 4. The figure highlights the common elements (concepts) that encode an observer’s view of the world and the different representations of given explanations. The model process is guided by two levels; situational awareness and situational conveyance. The situational awareness model, presented by Endsley’s in (?), is a widely used situation awareness model consisting of three consecutive stages:

1. 1.

   Perceived input: In this stage, the observer perceives the basic elements and their properties in the environment—object, quantity, quality, spatial and temporal information.

2. 2.

   Reasoning process: Based on the perceived inputs, the observer makes causal inferences between their beliefs (including the actor’s mental model) and goals, generates counterfactuals, and associates them with their preferences.

3. 3.

   Projection: This stage presents the observer’s predictions of future actions guided by their expected goals and the uncertainty level based on the understanding of the previous stages.

In this level, two types of reasoning occur due to the two mental model representations (?). The first type is stereotypical reasoning, in which the observer reasons about others’ mental states (what the observer would have done). An example from the data corpus is: “It might just do old classic seven gate [a game strategy].” The second type is empathetic reasoning, where the observer casts themselves into the actor’s mental model and reasons as they would (what the actor would have done). An example is: “This is exactly what I was talking about, you do something to try to force them.” In practice, the observer often contrasts both views in a single explanation—what they think should happen vs. what the actor is likely to do. For example: “It’s actually going to look for a run by here with this scan it looks like but unfortunately unable to find it with the ravager here poking away.” This was observed when explanations were provided for either a failed plan or a sub-optimal one toward achieving a goal. For simplicity, we assume a local perspective in this work, where the actor’s mental state is equal to the observer’s through the data coding and model implementation process.

The second level is a situational conveyance, where different explanations are formed and communicated by the observer. At this level of the model, the explaining process requires additional strategic knowledge. This includes reasoning about contrastive, conditional, temporal, and spatial cases of problem-solving tasks, allowing for more than just a causal representation of a given explanation.

Table 3 shows the different codes and concepts that emerged from across the two studies. The given explanations include factual and experiential knowledge (‘belief’), subjective likes and objective assessments (‘preference’), the desired state to be achieved (‘goal’), and possible future actions (‘plan’). When people explain others’ actions, they infer their goals to provide better explanations (?). When explaining a recognized goal, people infer the most relevant evidence from their belief state. Thus, we break down the belief concept into an ‘observational marker’, an observed precondition that most influences goal prediction. This concept applies not only to optimal behavior, measured by traditional efficiency metrics such as time and shortest route, but also to suboptimal behavior.

Since recognition problems activate counterfactual thinking (?), explanations of GR reasoning also include ‘counterfactuals’ — observational markers, plans, and goals. Explaining counterfactual plans implies having counterfactual goals where the actor has no plan to achieve them. We introduced an uncertainty code as we found that observers use words expressing uncertainty to indicate their confidence level. Additionally, as the problem is to explain goals, ‘goal’ and ‘counterfactual goal’ codes were also included. Finally, our data show the presence of different reasoning processes in the explanations. Observers associate a simple causal, conditional, contrastive, temporal, or spatial relationship when generating explanations. A combination of different concepts was used to form the given explanations.

Given that we conducted two distinct studies, we developed specific codes tailored to each. In the context of the Sokoban game, the code ‘counterfactual observational marker’ is explicitly used in the explanations. This code refers to observed preconditions that are against the counterfactual goals. However, observers in the StarCraft game did not incorporate this concept in their explanations, likely due to time constraints and the assumed expertise level of the StarCraft audience.

In the StarCraft game, observers added additional object properties, such as quality and quantity, to describe the characteristics and measured values of objects. This detailed level of description is attributed to the greater complexity of the StarCraft game compared to Sokoban. Furthermore, observers included general domain knowledge (referred to as ”belief”) in their explanations to clarify facts that were inaccessible to the audience due to the partial observability of the game domain.

Figure 5 illustrates the frequency distribution of 17 codes across two case studies. Given the navigational nature of both domains, observers predominantly referenced objects and their spatial properties to reflect the players’ strategies.

Among the various codes, the observational marker emerges as the most significant finding. This code, which ranks fourth in frequency, provides crucial insights into how observers infer players’ intentions and strategies. For instance, in the Sokoban game dataset, an observer noted, ”The player positioned itself on top of the box, leading me to believe it is going to push down on the box to reach goal 2.” Here, the action ”positioned itself on top of the box” is used to explain the entire observed sequence, forming a critical precondition for achieving the predicted goal. This demonstrates how a single observed action can be pivotal in understanding the player’s overall strategy. The counterfactual observational marker was coded only in the Sokoban game, as participants tended to use it when responding to ’why not’ questions. In the dynamic and time-constrained environment of live StarCraft commentary, shoutcasters were observed to focus heavily on observational markers in their explanations, even when addressing ’why not’ questions.

In addition, the third most frequently occurring code is the ‘plan’ code, where the actor’s goal is explained in terms of how future actions (plans) contribute to achieving that goal. This explanation, provided in terms of future actions, offers insights into potential actions and their execution (?). We believe that people tend to explain in terms of future actions as a way to resolve uncertainty. Causality is also frequently involved, as observers often associate causal relationships to generate explanations.

In human studies, two forms of causal reasoning are used to answer certain questions (?): retrospective reasoning involves explaining past events through counterfactual reasoning, which considers what could have happened if the observed facts were different, prospective reasoning involves explaining future events through transfactual reasoning, which considers what could happen in the future if certain conditions are met.

In the Sokoban game, we asked two questions: ‘Why?’ and ‘Why Not’, since it has been proved that they are the most demanded explanatory questions (?). ”Why” questions typically demand contrastive explanations, which are addressed through counterfactual reasoning (?). In such explanations, people answer ‘Why A’ in the form of ‘Why A instead of B?’, where B is some counterfactual goal(s) that did not happen. From the data collected, we classified the provided contrastive explanations into three categories: implicit, where observers implicitly contrast and identify relevant causes for A (the predicted goal(s)); explicit, where observers explicitly contrast and identify relevant causes for B (the counterfactual goal(s)); and extensive, where observers provide explanations for both by identifying relevant causes for A and also for B. It is important to note that the observer answered a “why” question in the ‘why’ condition, a “why not” question in the ‘why not’ condition, and both questions sequentially in the ’dual’ condition.

Figure 6 illustrates the differences between conditions. In the dual condition, observers tended to adopt an implicit mode when they answered the why question–after having answered why-not—more frequently compared to the why’ condition. A similar trend was observed for the why-not question in the dual condition, where observers tended to adopt an explicit mode when they answered the why-not question–after having answered why —more frequently compared to the Why-not condition. The contrastive nature of these explanations becomes particularly evident in the ’dual’ condition, where observers can differentiate between the two types of questions.

Participants primarily engaged in transfactual reasoning when they were highly uncertain about the agent’s goal. This uncertainty is a result of the agent’s observed behavior being suboptimal (irrational) to all goal hypotheses. For example, from the dataset: ”If the player keeps pushing the two boxes together, it would be impossible for box 2 to be put back onto a goal.”

The key codes highlight the important concepts used to explain goal recognition. Additionally, we coded implicit questions that shoutcasters tried to answer through their predictions.

In the StarCraft game, there are no pre-specified questions for the observers (shoutcasters) to answer. Instead, they gather information and craft explanations to address the audience’s implicit questions. We coded implicit questions that shoutcasters tried to answer through their predictions. Table 7 shows these questions and their frequencies. The shoutcasters are primarily focused on reasoning prospectively, addressing the ”What could happen?” question allows them to anticipate future events. Despite the time constraints, they managed to provide reasoning for their predictions by answering other questions of interest.

The most frequently answered question is the “Why?” question, involving retrospective reasoning over past events that mostly influence goal prediction. They further support their predictions by anticipating factors that control the context, allowing them to prospectively answer the “How?” question through future projections.

Causal reasoning is essential for constructing mental representations of events (?, ?, ?). These representations form causal chains that illustrate how a sequence of causes leads to an outcome. In the realm of explainable AI, an agent aiming to explain observed events may need to use abductive reasoning to identify a plausible set of causes (?). While numerous causes can contribute to an event, individuals typically select a subset they deem most relevant for their explanation (?).

Our human studies focus on understanding how people explain others’ predicted goals based on observed behavior. By coding these explanations, we identify a key concept, referred to as an ’observational marker’ that participants use to build their explanations. Our findings align with social and cognitive research indicating that people prefer explanatory causes that seem sufficient in the given context for the event to occur (?, ?, ?, ?).

People make causal inferences about others’ beliefs and goals based on their observed behavior and prior domain knowledge (?). A key aspect to explain those inferences is the ability to decide to what degree the observed evidence from a causal chain supports a goal hypothesis. To this end, we propose an explanation model for GR agents based on concepts such as causality and observational markers.

We focus on providing explanations for the output of the Mirroring GR algorithm (?, ?). However, our approach is agnostic of the underlying GR algorithm and will work for any GR algorithm that fits Definition 4.2. The Mirroring algorithmis inspired by people’s ability to perform online GR, originating from the brain’s mirror neuron system, which is responsible for matching the observation and execution of actions (?). The approach falls under the plan recognition as planning GR methods (?, ?) and uses a planner within the recognition process to compute alternative plans.

Specifically, the Mirroring algorithm uses a planner to compute optimal plans from an initial state $I$ to each goal $g_{j}\in G$ and to compute suffix plans from the last observation $o_{i}\in O$ to each goal $g_{j}\in G$. Observations are processed and evaluated incrementally. These suffix plans are then concatenated with a prefix plan (the observation sequence $O$ at time step $t$) to generate new plan hypotheses. The algorithm subsequently provides a likelihood distribution, i.e., posterior probabilities $P(g_{j}\mid O)$ for each $g_{j}\in G$ by evaluating which of the generated plans, incorporating the observations $O$, best matches the optimal plan. The following example illustrates The Mirroring approach to solving a GR task.

Answering why goal $g$? questions, such as Why was goal $g$ predicted as the most likely goal over all other alternatives?, rely on identifying the most important observation(s) that support the achievement of that goal. We call such observations observational markers (OMs).

It is generated for every possible alternative and in case of having multiple such actions, we select them all. Consider the navigational GR scenario presented in Figure 8. Let us answer the question Why $g_{2}$?. From the complete explanan of $g_{2}$, shown in Table  4:

After ranking them from highest to lowest, we obtain $\langle 0.85,o_{8}\rangle$ that has the highest value. This indicates that this observation is the $OM$, as in the observation that best explains the predicted goal hypothesis $G_{p}=\{g_{2}\}$ instead of the counterfactual goal hypotheses, $G_{c}=\{g_{1},g_{3}\}$. Therefore the explanation would be Because the agent has moved up from cell 26 to cell 17.

The question of why not $g^{\prime}$? relies on identifying the most important observation(s) related to $g^{\prime}$, which are called counterfactual observational markers.

There may be multiple such observations, in which case we select them all. Consider the navigational GR scenario (Figure 8) and the question Why not $g_{1}$ and $g_{3}$? From the complete explanation of $g_{1}$ and $g_{3}$, shown in Table  4:

After ranking them from lowest to highest, we obtain $\langle 0.28,o_{5}\rangle$ as the lowest value for $g_{1}$ and $\langle 0.18,o_{8}\rangle$ as the lowest value for $g_{3}$. This indicates that these observations are the counterfactualOM, the observations that best explain the counterfactual goal hypotheses, $G_{c}={g_{1},g_{3}}$. Therefore, the explanation would be: Because the agent moved right from cell 23 to cell 24, away from $g_{1}$, and up from cell 26 to cell 17, away from $g_{3}$.

We used the method of annotator agreement and ground truth, where human annotations of representative features provided the ground truth for quantitative evaluation of explanation quality (?).

We applied our XGR model to the online Mirroring implementation for each of the 15 scenarios. To determine the value of ($a_{XGR}$), we obtained the ranked explanation list from the XGR model based on the Weight of Evidence (WoE) values. For the question ‘Why goal $g$?’, we ranked the explanations in descending order of their WoE values, assigning a rank of 1 to the explanation with the highest WoE (Observational Marker, OM). Similarly, for the question ”Why not goal $g^{\prime}$?”, we ranked the explanations in ascending order of their WoE values, assigning a rank of 1 to the explanation with the lowest WoE (counterfactual OM). Using ground truth obtained through human annotation, we matched these ranks with their equivalents in the ranked explanation list for each question to determine $a_{groundTruth}$. We then calculated the Mean Absolute Error (MAE) for each question and across all 15 scenarios.

The Mean Absolute Error (MAE) is the average of the errors; hence, the larger the number, the larger the error. An error of 0 indicates full agreement between the models.

The results of the comparison are presented in Table  7. Each row shows the MAE calculated for each game scenario. The ‘Why’ and ‘Why not’ columns represent the MAE for our model compared to the human ground truth, while the CF(%) column represents the percentage of counterfactual action explanations that agree with the human ground truth.

For the majority of instances, the XGR model agreed with the ground truth obtained through human annotation. When answering ‘Why $g$?’ questions, the model had a full agreement with the ground truth in 11 out of the 15 scenarios (73.3%). For ‘Why not $g^{\prime}$?’ questions, the model had a full agreement with the ground truth in 14 out of the 15 scenarios (93.3%). By ‘full agreement’, we mean that the human annotators identify the same two actions as the primary explanation.

The CF column represents the percentage of counterfactual action explanations that agree with the human ground truth. Higher values are better, with 100% indicating full agreement with the ground truth counterfactual actions. The model achieved full agreement in 11 out of the 15 scenarios (73.3%), demonstrating excellent performance.

To better understand the performance of our model we investigated scenario 5 in game 3, which has a relatively high Mean Absolute Error (MAE) for the Why goal $g$? question, we refer to Figure 9. In this scenario, the agent delivers 2 boxes to 2 different locations and can push 2 boxes simultaneously. The blue arrows in the figure represent the observation sequence, indicating that the agent started at the blue circle and followed the arrows to its current location.

In this scenario, the most likely goal candidate, as predicted by the Mirroring GR algorithm, was delivering Box1 to $g_{1}$ and Box2 to $g_{2}$, i.e., $G_{p}={(g_{1},g_{2})}$. The counterfactual goal candidates involved delivering the boxes to either $g_{3}$ and $g_{4}$ or $g_{5}$ and $g_{6}$, with $G_{c}={(g_{3},g_{4}),(g_{5},g_{6})}$. It is important to note that to push both boxes simultaneously to goal $(g_{1},g_{2})$, the agent would need to stand to the right of the boxes. Conversely, to push both boxes simultaneously to either $(g_{3},g_{4})$ or $(g_{5},g_{6})$, the agent would need to position itself to the left of the boxes.

The XGR model’s explanation for the Why goal $g$? question is highlighted by the green, hollow circle in the figure. This observation suggests that the agent aims to position itself to the right of both boxes, confirming the hypothesis that the goal is $(g_{1},g_{2})$. Considering the agent’s ability to push multiple boxes, this observation constitutes the Observation Model (OM) with the highest Weight of Evidence (WoE).

On the other hand, the annotators established the ground truth explanation by choosing the second observation, $o_{2}$, for both the Why goal $g$? and Why not $g^{\prime}$? questions. According to our model, this observation is the one with the lowest Weight of Evidence (WoE), actually making it the counterfactual observation and the answer to the question Why not $g^{\prime}$? This is because this observation moves away from both goals $(g_{3},g_{4})$ and $(g_{5},g_{6})$.

Participants choosing to use the same answer for both Why goal $g$? and Why not $g^{\prime}$? questions can also be found in other instances of discrepancies between the output of our model and the ground truth. The difference in explanations can be attributed to some confusion and/or preference between why? and why not? questions on the part of the participants.

To address this, we conducted a follow-up experiment where we presented the participants with the scenarios they were confused with (Table 7, scenarios with bold values). For each scenario, we provided them with explanations from two systems: the first system’s explanation from our model, and the second system’s explanation from the ground truth. We then asked them which system provided a better explanation. All three participants preferred the explanations provided by our model which leads us to our model’s explainability as perceived by users.

We first discuss the results of the first hypothesis, where we investigate whether our model leads to a more accurate decision.

For the second hypothesis, we evaluate how quickly participants made their decisions, hypothesizing that our model will contribute to greater overall efficiency in task completion. Efficiency was measured by the overall time spent on the tasks, recorded in seconds. statistical analysis revealed significant differences in completion time across the four conditions ($\chi^{2}=39.28$, p $<0.001$). Post hoc pairwise comparisons revealed that participants in the explanation cohorts, XGR_WoE and XGR_WoE_CF, had significantly higher completion time compared to participants in the NoGR and GR cohorts ($p<0.001$). As shown in Figure 16, there are significant differences in completion time across the four conditions. However, no statistically significant difference was found between NoGR and GR ($p=0.13$), nor between XGR_WoE and XGR_WoE_CF ($p=0.15$). Therefore, we reject the hypothesis that our models improve overall task efficiency.

For the third hypothesis, we examine whether our model influences the appropriateness of human reliance, specifically by reducing overreliance and underreliance on the GR model. We measure overreliance as the fraction of incorrect human decisions that align with incorrect GR predictions, calculated over the total number of incorrect GR predictions. Conversely, underreliance is measured as the fraction of incorrect human decisions that occur despite correct GR predictions, calculated over the total number of correct GR predictions.

We now report the results of our evaluation to determine whether our explanation model promotes trust in the GR agent. Figure 19 illustrates the distribution of Likert scale data across conditions. The statistical test was performed, yielding a significant result ($p<0.001$) for all metrics, indicating a significant difference among the three conditions for the Confident, Predictable, Reliable, and Safe metrics. Subsequently, we conducted pairwise comparison tests, and the results are summarized in Table LABEL:tab:trustscale. These findings highlight that the XGR_WoE_CF condition shows significant results across all metrics. Therefore, we accept the hypothesis that our model, XGR_WoE_CF, effectively promotes trust in the GR agent.

We now report the results of the self-reported metrics of explanation satisfaction. We performed statistical tests to examine whether there are any significant differences between the explanation models XGR_WoE and XGR_WoE_CF for the explanation quality metrics: understanding, satisfying, sufficient detail, and completeness. The obtained p-values ($0.20,0.31,0.34,0.24$) for these four metrics indicate no significant differences between the two models. Although the differences are not statistically significant, Table 12 shows that the XGR_WoE_CF model is generally better perceived across these metrics, with more consistent responses from participants. This mirrors the results in decision accuracy, as participants feel they have a better understanding of the agent.

Participants answered an open-ended question about their decision justification in the last scenario. After excluding answers with fewer than three words or containing gibberish, a total of 244 textual data points were collected. We focused on understanding the reasoning process and explanation concepts used across the four conditions. We systematically coded the responses following a hybrid approach of deductive and inductive thematic analysis approach (?), labeling responses based on our pre-established main concepts (observational marker and counterfactual observational marker). We then analyzed participants’ responses and identified a few differences in their justifications across the four conditions.