Control-Theoretic Analysis of
Shared Control Systems

In a shared control framework, both the user and an assistive algorithm simultaneously control a robot [4]. The assistance modifies the robot’s behavior to drive the system in a way that the user “actually means” — motivations for shared control include challenges of low-dimensional interfaces [5, 6] or safety-critical situations [7]. Goal-predictive assistance [2] breaks the problem down into intent detection, during which the system estimates the “true” goals of the user using various methods[8, 9, 10], and arbitration, during which the user’s initiative is mixed with an assistive action generated based on the detected intent. These systems improve performance in many settings [11, 12, 13, 14, 15]. However, they require strong assumptions about user behavior which do not necessarily hold in practice [1], thus benefiting from our approach.

Other assistance methods focus directly on the dynamics. Most directly, virtual fixtures [16] apply artificial forces to the robot’s input response. More sophisticated approaches change the behavior of the robot to encourages successful task performance [17, 18]. Our analysis broadens this assistance strategy to characterize assistance behavior that is not designed this way explicitly.

Though shared control research generally focuses on algorithmic advances, several papers consider the human experience of controlling a robot in a shared control paradigm [19, 1]. This work includes explicitly modeling how the user will adapt to the system [20, 21], changing the system to make the user’s behavior more informative [22, 23], and adjusting the user model based on observed behavior [14, 24, 25].

Control theory analyzes how to drive a dynamical system to a particular desired output. Dynamical systems models consist of three components: the input $u$, the state $x$, and the output $y$. Then, control theory answers the following question: Assume we can provide inputs $u$ to the system and observe output $y$, but we do not have direct access to $x$. How do we select the input $u$ to drive the system to produce some desired output $y^{*}$? For this paper, we examine assistance from the perspective of the user, who can provide input $u$ to the assistance system and observe output $y$ but does not have direct access to the internal state $x$. We then use insights from control theory about what makes dynamics easy or hard to control and apply them to designing the assistance itself.

Control theory techniques are common in robotics in general. Most low-level motor controllers are PID (proportional-integral-derivative) systems, while model-predictive control is used for short-horizon optimization of robot low-level commands. Here, we consider the inverse problem: how do we design the dynamics of a system so that it is easy for a controller to be designed that drives the system to an intended goal?

We begin by briefly summarizing the shared autonomy algorithm presented in Javdani et al. [26]. The algorithm expects a pre-specified (finite) list of $k$ goals $G$ along with optimal policies for each goal defined by action-value functions $Q_{g}(x,a)$. It receives an input $u_{t}$ from the user at each time step, uses a Bayesian update framework to estimate the user’s true goal within $G$ as a probability distribution, then takes an action that is optimal in expectation over the probability of the user’s true goal. This assistive action is then applied to the system itself. The algorithm is intended to improve performance of the system for a user who knows their goals but cannot optimally drive the robot to achieve them; the robot uses its own (presumably) optimal policy to reach them instead. This system decreases task completion time and user effort required for reaching the goal.

At each time step, the system estimates the likelihood of an observed user action $u_{t}$ given each goal. Several models for the likelihood are used [9, 10]; here, we will show the Bolzmann rationality formulation, which sets

$$p(u_{t}|x_{t},g)\propto\exp\beta Q_{g}(x_{t},u_{t})$$

(1)

with $x_{t}$ the current state, $\beta$ a “rationality” parameter that varies from $\beta=0$, indicating random user actions, to $\beta=\infty$, indicating perfect user actions. Note that this probability is conditioned on the stat $x_{t}$; this approach treats the user’s input as informative separately from the actual robot state in which it was provided. For ease of notation, we drop $x_{t}$ throughout.

Next, this likelihood is used to update the assistance system’s estimate of the user’s goal, stored as a probability distribution $p(g|u_{0},\cdots,u_{t})$. Via Bayes’ rule, we can set

$$p(g|u_{0},\cdots,u_{t})=\frac{1}{Z_{t}}p(u_{t}|g)p(g|u_{0},\cdots,u_{t-1}),$$

(2)

with $Z_{t}=\sum_{g}p(g|u_{0},\cdots,u_{t})$ a normalization factor.

Finally, this system generates assistance by selecting an action $a_{t}$ that has the largest expected increase in value over the goal distribution, given by

$$a_{t}=\operatorname*{arg\,max}_{a^{\prime}}\sum_{g}p(g|u_{0},\cdots,u_{t})Q_{g}(x_{t},u_{t}).$$

(3)

This equation is derived from applying the QMDP assumption to solving a POMDP representation of the user’s behavior; see the paper for further details.

We now show how the same assistance system appears to the user as a dynamical system. To do so, we define the internal state of the system and its dynamics, how the state changes based on the input, and how the output of the system is derived from the state.

First, for convenience, we define $\vec{Q}_{t}(u)=[Q_{g_{0}}(u),Q_{g_{1}}(u),\cdots,Q_{g_{k-1}}(u)]^{\mkern-1.5mu\mathsf{T}}$ as a vector-valued function representing the value of taking the same action $u$ in the state $x_{t}$ with respect to each of the goals in turn. This transformation turns a possible action $u$ into a vector of length $k$ that describes how well $u$ aligns with each goal in turn.

Next, we define the state of the assistance system $\ell_{t}$ as

$$\ell_{t}=[\log p(g_{0}|u_{0},\cdots,u_{t}),\cdots,\log p(g_{k-1}|u_{0},\cdots,u_{t})],$$

(4)

the log of the goal probability vector used in each step. By taking the log of Eqn. 2 and substituting $\ell_{t}$ as appropriate, we obtain the system dynamics

$$\ell_{t}=\ell_{t-1}+\log p(u_{t}|g)-\log Z_{t}.$$

(5)

Note that this is a linear system in $\ell$, but is nonlinear in the user input $u_{t}$. However, the user contribution to the update is in a single additive term, so we can perform a change in variables to remove it. We define $v_{t}(g)=\log p(u_{t}|g)$ as contribution of the user’s input towards the system’s selection of goal $g$; using Eqn 1, we compute

$$v_{t}(g)=\beta Q_{g}(u_{t})-\log\sum_{u^{\prime}\in U}\exp(\beta Q_{g}(u^{\prime})).$$

(6)

Using this equation, we can express the dynamics in terms of the input vector $\vec{v}_{t}=[v_{t}(g_{0}),v_{t}(g_{1}),\cdots,v_{t}(g_{k-1}]$ as

$$\ell_{t}=\ell_{t-1}+\vec{v}_{t}-\log Z_{t}.$$

(7)

Next, we consider the output of the system $a_{t}$. Using Eqn. 3 and substituting $\ell_{t}$, we find

$$a_{t}=\operatorname*{arg\,max}_{a^{\prime}}\exp\ell_{t}\cdot\vec{Q}(a^{\prime}).$$

(8)

Now, note that the term $Z_{t}$ is positive, uniform over $g$, and only used inside the $\operatorname*{arg\,max}$ term, so it can be dropped entirely (though normalization may be required for numerical stability). Thus, our final system is

	$\displaystyle\ell_{t}$	$\displaystyle=\ell_{t-1}+\vec{v}_{t}$
	$\displaystyle a_{t}$	$\displaystyle=\operatorname*{arg\,max}_{a^{\prime}}\exp\ell_{t}\cdot\vec{Q}_{t}(a^{\prime}).$

The system dynamics for shared autonomy are relatively simple. Though (generally) nonlinear functions $v_{t}(g)=v(x,u,g)$ are required to process the raw user input $u_{t}$ and compute the output action $a_{t}$, the system dynamics consists of a single integrator.

This representation gives us several insights into the behavior of the assistance system from the perspective of the user. Pure integrator systems are known to have infinite DC gain: if supplied a constant input, the system will increase without bound. In this case, providing a constant user command will cause the probabilities assigned to goals other than the most likely to decrease without bound. If the user has been pursuing one goal for some time and then switches to a different goal, the user must drive the system towards the new goal with an amount of effort proportional to what was spent achieving the first goal before the system will register a change in their intention. Though the original formulation of shared autonomy assumes that users will not change their goals, we can still analyze the system behavior if they do.

Dynamical systems theory also gives us insight into how to mitigate this effect. Since we have full control over the dynamics of the goal update, we can introduce a new term to Eqn. 2 that adds internal stability to the system:

$$\ell_{t}=k\ell_{t-1}+v_{t},$$

(9)

where $0\leq k\leq 1$ is a stabilizing term that limits how extreme the values of $\ell_{t}$ can become. Given a consistent user input, the probability distribution will eventually reach a steady-state value (ignoring the effect of renormalization), so the effort to change the system to select a different goal remains bounded. Control theory offers other options as well, such as adding a term dependent on $\vec{v}_{t}-\vec{v}_{t-1}$ which monitors the change in user input. The analysis suggests changes in the dynamics of the system that the user is operating which can be evaluated in future user studies.

Now, we turn to a different question: what are the possible actions that an operator can cause the system to produce? This question makes no sense in a robot-centric perspective: the goal-specific policy defines the optimal actions for each goal, so no additional actions are needed. However, users may want to drive the system to produce other actions based on their individual information or preferences, and we can use similar analysis tools to determine their options. To examine the possible actions, we consider Eqn. 3. For the system to select an action $a$, the action must be the $\operatorname*{arg\,max}$ for some value of $\ell_{t}$. Furthermore, since $\exp\ell_{t}>0$, the term inside the $\operatorname*{arg\,max}$ is a positive combination of $\vec{Q}$; by definition, then, $\vec{Q}_{t}(a)$ must lie on the Pareto frontier of $\{\vec{Q}_{t}(a):a\in A\}$. Therefore, not all actions $A$ are possible. In fact, the possible actions are a function only of the state $x_{t}$. User input can only select among the Pareto-optimal actions, and other actions are impossible to achieve no matter the user input (see Fig. 1).

With simplifying assumptions, we can go further. Consider a free-space navigation system with real states $S=\mathbb{R}^{n}$ and real unit actions $A=\{a\in\mathbb{R}^{n}:|a|=1\}\cup\{0\}$; we will denote these states and action as $\vec{x}$ and $\vec{a}$ for clarity. Further, let each goal $g\in S$ indicate a target state, which we will denote $\vec{g}$; $G$ represents a matrix $[g_{0},g_{1},\cdots,g_{k-1}]$ of all goals with each row corresponding to a goal. We posit a closed-form value function $Q_{g}(\vec{x},\vec{a})=(\vec{g}-\vec{x})^{\mkern-1.5mu\mathsf{T}}\vec{a}$ which computes how well the proposed action $\vec{a}$ aligns with the direction to the goal $(\vec{g}-\vec{x})$; with this assumption, the system is a discrete-time analogue of potential function navigation¹¹1On the domain $|g-x|>1$, with $V(x)=|g-x|$, $r(x)=(g-x)^{\mkern-1.5mu\mathsf{T}}a-|g-x-a|$, and $T(x,a)=x+a$, this $Q$ satisfies the Bellman equation.. Then, $\vec{Q}(a)=Q^{\mkern-1.5mu\mathsf{T}}\vec{a}$ is in fact a matrix multiplication with

$\displaystyle Q=\begin{bmatrix}|&|&&|\\ \vec{g}_{0}-\vec{x}&\vec{g}_{1}-\vec{x}&\cdots&\vec{g}_{k-1}-\vec{x}\\ |&|&&|\\ \end{bmatrix}.$

(10)

In this scenario, we can solve the $\operatorname*{arg\,max}$ term directly:

	$\displaystyle\vec{a}^{*}$	$\displaystyle=\operatorname*{arg\,max}_{\vec{a}}\exp\ell\cdot\vec{Q}(\vec{a})$
		$\displaystyle=\operatorname*{arg\,max}_{\vec{a}}(\exp\ell)^{\mkern-1.5mu\mathsf{T}}Q^{\mkern-1.5mu\mathsf{T}}\vec{a}$
		$\displaystyle=\frac{Q\exp\ell}{|Q\exp\ell|}.$

Thus all assistance actions $\vec{a}$ are normalized positive combinations of the columns of $Q$, i.e., within the positive span of the directions towards each goal.

From this analysis, we can draw several conclusions. First, the assistance can produce any action in $A$ precisely when the positive span of $Q$ includes the whole space. This scenario occurs when the origin lies within the convex hull of $Q$ or, equivalently, when the state $\vec{x}$ is inside the convex hull of the goals $\vec{g}$. Thus, the assistance dynamics have two phases: when the state is outside the convex hull of the goals, the assistance will drive the state towards the convex hull no matter the user input. The user can change the direction of approach among the available goals but cannot drive the system away from the convex hull. Once the system is within the convex hull, any action can be achieved by manipulating the assistance state $\ell_{t}$ so long as the system state remains within the convex hull; the user cannot drive the system outside the hull. This hull always exists because $|G|$ is finite (by assumption), though it can be degenerate; in this case, the system is driven to and remains in the hyperplane defined by the goals and by the convex hull within that hyperplane.

Another result of this analysis is that the system can be made to stop at any point within the convex hull of the goals. Formally, for all states $\vec{x}\in\text{conv}(\vec{g})$, there is some vector $v$ with all nonnegative elements such that $Q\,v=0$. Since $v$ is nonnegative, we can select $\exp\ell=v$; for this probability state, the optimal action $a=\text{norm}(Q\exp\ell)=0$. Conversely, all goal prediction states have a unique stable point at $\vec{g}\exp\ell=\sum_{k}p(k)g_{k}=x^{*}$, the convex combination of the goal locations with weights equal to the goal probabilities; this fact follows from solving $Q\exp\ell=0$ for $\vec{x}$.

Putting everything together, shared autonomy imposes a two-phase system. When the system state is outside the acceptable region, defined by the (possibly lower-dimensional) convex hull of the goals, the system will drive it towards the convex hull independent of user input. Within the convex hull, though, the system supports any desired action, but through a layer of indirection. The user input updates the set point $x^{*}$ via the goal inference step, and then the system drives the state towards that set point. Within the convex hull, users cannot control the robot directly; rather, they move this (invisible) set point, and then system responds. Controlling this system is like driving a car and controlling its speed only through cruise control. The layer of indirection may also explain why users remain unsatisfied with the system despite improvements in trial metrics [3] and particularly dislike pure assistance without any arbitration with the user’s raw input [27]. This analysis also explains the equilibrium that users discovered intuitively in Aronson and Short [1]: by carefully altering the set point, users can cause the system to stop at any given location within the convex hull of the goals.