Dynamic Competency Self-Assessment for Autonomous Agents

To prevent unnecessary assessments and save onboard computational resources, we take an event triggered approach to in situ self-assessment. Our algorithm assesses confidence only when there is evidence that the agent’s task confidence has changed.

One promising method for detecting such a change is the Surprise Index (SI). SI is defined as the sum of probabilities of more extreme (or less probable) events than an observed event given a probabilistic model (Zagorecki et al., 2015). For a given event $e\in E$, SI is computed by summing over the probabilities of more extreme events in the distribution $p(E)$:

SI can be though of as how (in)compatible an observation $e$ is given a set of possible events $E$. This is a similar to the more well known entropy based surprise (Baldi and Itti, 2010; Benish, 1999); however entropy based surprise is unbounded, while Surprise Index is bounded between zero (most surprising) and one (least surprising). SI also shares similarities with the tail probability or the p-value given the hypothesis that $e$ is from the distribution $p(E)$; a large p-value (large SI) indicates strong evidence that $e$ is likely from $p(E)$, while a small p-value indicates strong evidence to the contrary.

In this work we are interested in determining when the agent should re-assess its task confidence. We propose computing the SI of the agent’s observed state $s_{t}$ with respect to that agent’s model prediction $p(\hat{s}_{t})$, and triggering a re-assessment when the SI falls below a threshold $\delta$. In essence, we are monitoring the quality of the agent’s model given the task and triggering an assessment when quality wanes. Because some aspects of $s_{t}$ may be more relevant to competency than others, we compute SI over a subset of the state marginals.

To assess competency we leverage the Generalized Outcome Assessment (GOA) (Conlon et al., 2022a). Given a probabilistic world model $M$, GOA simulates task execution by rolling out state predictions $p_{M}(s_{t+1}|s_{t},a_{t})$. Note that a $M$ could take the form of a Monte Carlo based planner(Israelsen, 2019), a black box neural network (Conlon et al., 2022a; Ha and Schmidhuber, 2018), or similar. GOA then analyses the state predictions and computes the agent’s margin of confidence in attaining an outcome better than some target outcome threshold $Z$. Examples of target outcomes could include craters hit (which we prefer less of) or packages delivered (which we prefer more of). The confidence value can be reported as a raw value $\in(0,1)$ or mapped to a semantic label indicating confidence such as highly likely, likely, unlikely, highly unlikely. For the experiments outlined later, we use the raw numerical values of confidence.

We call our method for surprise-based dynamic self-assessment Event-Triggered Generalized Outcome Assessment (ET-GOA). The algorithm is presented in Alg. 1 and can be broken up into two components: (1) before task execution and (2) during task execution.

Our experimental scenario was based on the motivating SAR example from section 1: A single agent was tasked to safely deliver cargo to one of three goals. The environment contained two types of obstacles: craters and dust zones, which were difficult for the agent to avoid. Driving over craters damaged the agent, and if enough craters were hit while navigating then the agent was considered broken and failed the delivery task. Dust zones degraded sensors and injected noise into the agent’s state transition dynamics. Dust zones were generally found near craters, which increases the chance that the agent hit a craters if it found itself in dust. To simulate environmental changes that would occur in realistic deployments, new obstacles could spawn at random locations (except for the agent’s location) during task execution.

The environment was a custom OpenAI Gym environment (Brockman et al., 2016). The agent was modeled as a discrete state/action Markov Decision Process with state space $s=(s_{x},s_{y},s_{c},s_{z})$ consisting of the agent’s $(x,y)$ location and the counts of craters $(s_{c})$ and dust zones $(s_{z})$ within its sensor field of view (FOV). The sensor FOV was modeled as omnidirectional with a radius of 10 grid squares. The total size of the 2D environment was 50x50 grid squares. We trained one policy for each goal using Q-Learning (Watkins and Dayan, 1992). No obstacles were present during training to prevent the agent from learning how to overcome the difficulties of the environment.

The world model $M$ used for self-assessment was a copy of the environment that had an identical state transition function but only included known craters and dust zones. The agent chose the goal which had the maximum assessed confidence. If there was a tie in confidence, the agent chose the closest goal. An example environment can be seen in Fig. 1.

We evaluated two different environments, static and dynamic. In the static environment, the locations of craters and dust zones were known by the agent a priori and remained unchanged for the entire task execution. In the dynamic environment, the locations of craters and dust zones were initially known, but changed at a predetermined time to simulate a previously generated onboard navigation map suddenly becoming out-of-date.

We had three hypotheses: (1) In a static environment, ET-GOA and GOA will perform equally well and will both outperform a random goal choice; (2) In a dynamic environment, ET-GOA will outperform both GOA and random choice; (3) ET-GOA can capture both positive and negative changes in task difficulty. We analyzed agent performance (number of deliveries) for hypothesis 1 and 2, and we analyzed reported confidence relative to task difficulty changes for hypothesis 3.

Our first experiment was used to validate our first two hypotheses. At $t=0$ we initialized the agent with the locations of all obstacles. For dynamic conditions we changed the locations of the obstacles without the agent’s knowledge at $t=10$. Three conditions were considered: no assessment, GOA, and ET-GOA. The no assessment condition did not use any competency assessment. Rather, at $t=0$ the agent chose the goal at random and navigated directly to it. The GOA condition used the standard Generalized Outcome Assessment analysis discussed in (Conlon et al., 2022a). At $t=0$ the agent selected and navigated to the goal $g\in G$ with the highest GOA confidence according to Eqn. 2.

The ET-GOA condition used the ET-GOA algorithm discussed in 3.3. We used two state marginals as triggers: $s_{c}$ (and $\hat{s}_{c}$), the actual (and predicted) craters visible in the agent’s FOV; and $s_{z}$ (and $\hat{s}_{z}$), the actual (and predicted) dust zones visible in the agent’s FOV. This essentially computed the surprise between the expected obstacle locations (from the initial location information) and the “on the ground” obstacle locations observed while traversing the environment. We chose these specific marginals because they align with the sensor capability in modern robots. Additionally, looking at surprising observations in the agent’s sensor FOV enabled the algorithm to trigger a re-assessment prior to the agent physically coming into contact with a possibly dangerous obstacle. The algorithm triggered a re-assessment if the minimum SI of either state marginal was less than $\delta=0.05$:

The agent then selected the goal based on eqn. 2. The agent navigated directly to that goal until it either reached the goal or triggered a re-assessment and chose a new goal. For each condition the agent attempted 100 delivery tasks.

Our second experiment was used to validate our third hypothesis. Here we evaluated at how well ET-GOA captured changes in the environment that impacted task difficulty. For this evaluation, we isolated the agent to navigate to a single static goal location under two conditions. The first, called $easy\xrightarrow{}hard\xrightarrow{}easy$, starts out with no obstacles, then obstacles are randomly added at time step 10, then all obstacles are deleted at time step 30. The second, called $hard\xrightarrow{}easy\xrightarrow{}hard$, starts out with randomized obstacles, then all obstacles are deleted at time step 10, and then new obstacles are added at time step 30. Adding obstacles increases task difficulty for the agent and vice versa for deleting obstacles. Obstacles at time step zero were known to the agent, while the obstacles added/deleted at time steps 10 and 30 had to be observed in situ. We ran 100 episodes for each condition and recorded the initial assessment and the ET-GOA assessment after each add/delete event.