$\mathscr{APIA}$: An Architecture for Policy-Aware Intentional Agents

The Architecture for Intentional Agents, $\mathscr{AIA}$ [7], builds upon the Observe-Think-Act control loop of the AAA architecture⁶⁶6AAA stands for “Autonomous Agent Architecture.” [3] and extends it in a couple of directions. First, $\mathscr{AIA}$ adds the possibility for action failure: the agent attempts to perform an action during its control loop but may find that it is unable to do so. In this case, the action is deemed non-executable. Second, $\mathscr{AIA}$ addresses a limitation of AAA in which plans are not persisted across iterations of the control loop. In $\mathscr{AIA}$, agents pursue goals by persisting in their intentions to execute action plans known to satisfy these goals (i.e., activities). Activities are represented via a set of statics [6, 7]:

$$\begin{array}[]{l}\{activity(M),\ \ length(M,L),\ \ goal(M,G),\\ \ \ component(M,1,C_{1}),\ \ component(M,2,C_{2}),\ \ \dots,\ \ component(M,L,C_{L})\ \}\\ \end{array}$$

where $M$ is a unique identifier for the activity; $C_{1},C_{2},\dots,C_{L}$ are the $1^{st},2^{nd},\dots,L^{th}$ components of the activity; and $G$ is the goal that $[C_{1},C_{2},\dots,C_{L}]$ achieves. Some activities are pre-computed and stored in the agent’s knowledge base, while the rest can be generated on demand.

In addition to fluents and actions describing the agent’s environment, $\mathscr{AIA}$ introduces mental fluents to keep track of the agent’s progress in the currently intended activity and towards its desired goal. Mental fluents are updated through mental actions. The Theory of Intentions is a collection of axioms that maintain an agent’s mental state. Elements of the agent’s mental state include the currently selected goal $G$, stored in the $active\_goal\left(G\right)$ inertial fluent, and the current planned activity, stored in the $status(M,k)$ inertial fluent. When either a goal is selected or an activity is planned, they are said to be intended. Mental action $start(M)$ initiates the agent’s intention to execute activity $M$ and $stop(M)$ terminates it. Though most actions are executed by the agent itself, some must be executed by the agent’s controller. The exogenous mental action $select(G)$ causes the agent to intend to achieve goal $G$ while $abandon(G)$ causes the agent to cease its intent to achieve goal $G$.

An agent in the $\mathscr{AIA}$ architecture performs the following loop:

$$\begin{array}[]{l}\mbox{1. Interpret observations.}\\ \mbox{2. Find intended action }A.\\ \mbox{3. Attempt to perform }A\mbox{; record this attempt in history.}\\ \mbox{4. Observe the world; record observations in history.}\\ \mbox{5. Repeat (i.e.~{}go to step 1).}\end{array}$$

(1)

In real-world applications, an autonomous agent may be required to follow certain rules or ethical constraints, and may be penalized when acting in violation of them. Thus, it is necessary to discuss policies for agent behavior and a formalism with which agents can deduce the compliance of their actions. Gelfond and Lobo [13] introduce the Authorization and Obligation Policy Language $\mathscr{AOPL}$ for policy specification. An authorization policy is a set of conditions that denote whether an agent’s action is permitted or not. An obligation policy describes what an agent must do or must not do. $\mathscr{AOPL}$ works in conjunction with a dynamic system description written in an action language such as $\mathscr{AL}$. An agent’s policy is the subset of the trajectories in the domain’s transition diagram that are desired by the agent’s controller.

Policies of $\mathscr{AOPL}$ are specified using predicates $permitted$ for authorization policies, $obl$ for obligation policies, and static laws similar to those from action language $\mathscr{AL}$:

$$\begin{array}[]{rcl}permitted\left(a\right)&\textbf{ if }&cond\\ \neg permitted\left(a\right)&\textbf{ if }&cond\\ obl\left(h\right)&\textbf{ if }&cond\\ \neg obl\left(h\right)&\textbf{ if }&cond\end{array}$$

where $a$ is an action; $h$ is a happening (i.e., an action or its negation⁷⁷7If $obl(\neg a)$ is true, then the agent must not execute $a$ in the current state.); and $cond$ is a, possibly empty, conjunction of fluents, actions, or their negations. In addition to these strict policy statements, $\mathscr{AOPL}$ supports defeasible statements and priorities between them as in:

$$\begin{array}[]{lccl}d:\textbf{normally }&permitted(a)&\textbf{ if }&cond\\ d:\textbf{normally }&\neg permitted(a)&\textbf{ if }&cond\\ d:\textbf{normally }&obl(h)&\textbf{ if }&cond\\ d:\textbf{normally }&\neg obl(h)&\textbf{ if }&cond\\ &prefer(d_{i},d_{j})&&\end{array}$$

(2)

Gelfond and Lobo define policy compliance separately for authorizations vs. obligations:

Note that a set of actions $A$ can be strongly, weakly, or non-compliant with an authorization policy, but $A$ can only be compliant or non-compliant with an obligation policy. Given an $\mathscr{AOPL}$ policy (with authorization and obligation policy statements), $A$ is strongly compliant if it is strongly compliant with its authorization policy and compliant with its obligation policy. Likewise for weak compliance and non-compliance. Computing policy compliance is reduced to the problem of finding answer sets of a logic program obtained by translating the policy rules into ASP. In this translation, predicates $obl$ and $permitted$ are extended to include an extra argument $I$ standing for the time step, while $lp(x)$ denotes the ASP transformation of $x$, where $x$ can be a rule, an action literal, a fluent literal, or a condition.

$\mathscr{AOPL}$ does not discuss interactions between authorization and obligation policies on the same action, does not define compliance in terms of obligation policies for a trajectory in the dynamic system, and does not compare the degree of compliance of two trajectories. All of these aspects are relevant when modeling a policy-aware intentional agent and are addressed in our work.

Gelfond and Lobo [13] conceived $\mathscr{AOPL}$ as a means to evaluate policy compliance in a dynamic system. This differs from $\mathscr{AIA}$ in the following ways:

• •

  $\mathscr{AOPL}$ evaluates trajectories in a domain’s transition diagram from a global perspective whereas $\mathscr{AIA}$ distinguishes between agent actions and exogenous actions, and chooses which agent actions to attempt next.

• •

  $\mathscr{AOPL}$ evaluates histories at “the end of the day” whereas the $\mathscr{AIA}$ architecture, while still reasoning over past actions in its diagnosis mode, places an emphasis on planning future actions to achieve a future desired state.

These differences prevent $\mathscr{AOPL}$ policies from interoperating with the $\mathscr{AIA}$ agent architecture out of the box. To address the first issue, we constrain $\mathscr{AOPL}$ policies to describe only agent actions in our $\mathscr{APIA}$ architecture. For the second issue, we adjust our policy compliance rules such that only future actions affect policy compliance. Since our focus is on planning, in $\mathscr{APIA}$ past actions are always considered “compliant” although they might not have been at the time. For an agent that previously had no choice but a non-compliant action, this allows the agent to conceive of “turning a new leaf” and seeking policy-compliant actions in the future.

Also, $\mathscr{AOPL}$ does not include specification on how authorization policy statements interact with obligation policy statements. For example, consider the following $\mathscr{AOPL}$ policy:

$$\begin{array}[]{c}permitted(a)\\ obl(\neg a)\end{array}$$

which is contradictory, since the agent is permitted to perform action $a$ but at the same time is obligated to refrain from it. Appealing to common sense, if an agent is obligated to refrain from an action, one would conclude that the action is not permitted. Likewise, it makes sense to say that, if an agent is obligated to do an action, then it must be permitted. Thus, we take these intuitions and create the following non-contradiction ASP axioms, in which we use literals $obl$ and $permitted$ expanded to include a new argument $I$ representing the time step:

$$\begin{array}[]{ll}\leftarrow&agent\_action(A),\ \ obl(A,I),\ \ \neg permitted(A,I).\\ \leftarrow&agent\_action(A),\ \ obl(neg(A),I),\ \ permitted(A,I).\end{array}$$

(3)

These enforce that, at the very least, the authorization and obligation policies do not contradict each other, while allowing for defeasible policies to work appropriately.

We also extend the translation of defeasible policy statements. Suppose we have the following:

$$\begin{array}[]{c}\textbf{normally }permitted(a)\\ obl(\neg a)\ \textbf{ if }\ cond\end{array}$$

(4)

Using Gelfond and Lobo’s approach [13], the corresponding ASP translation would be:

$$\begin{array}[]{l}permitted(a,I)\leftarrow\textbf{not }\neg permitted(a,I).\\ obl(neg(a),I)\leftarrow lp(cond).\end{array}$$

where $lp(cond)$ and $obl(neg(a),I)$ represent the logic programming encoding of $cond$ and $obl(\neg a)$, respectively. Based on policy (4), both $permitted(a,I)$ and $obl(neg(a),I)$ would be true at a time step $I$ when $cond$ is met. This violates the non-contradiction axioms in (3). So, we replace the translation of the defeasible statement in (4) with the following encoding:

$$\begin{array}[]{lll}permitted(a,I)&\leftarrow&\textbf{not }\neg permitted(a,I),\ \ \textbf{not }obl(neg(a),I).\end{array}$$

This allows the presence of $obl(\neg a)$ to be an exceptional case to the defeasible rule.

In general, we propose translating the different types of defeasible statements in (2) as follows, respectively, where predicate $ab$ facilitates dealing with possible additional (weak) exceptions:

$$\begin{array}[]{rll}permitted(a,I)&\leftarrow&lp(cond),\ \ \textbf{not }ab(d,I),\ \ \textbf{not }\neg permitted(a,I),\ \ \textbf{not }obl(neg(a),I).\\ \neg permitted(a,I)&\leftarrow&lp(cond),\ \ \textbf{not }ab(d,I),\ \ \textbf{not }permitted(a,I),\ \ \textbf{not }obl(a,I).\\ obl(a,I)&\leftarrow&lp(cond),\ \ \textbf{not }ab(d,I),\ \ \textbf{not }\neg obl(a,I),\ \ \textbf{not }\neg permitted(a,I).\\ obl(neg(a),I)&\leftarrow&lp(cond),\ \ \textbf{not }ab(d,I),\ \ \textbf{not }obl(neg(a),I),\ \ \textbf{not }permitted(a,I).\end{array}$$

The $\mathscr{AIA}$ architecture, which is the underlying basis of $\mathscr{APIA}$, introduces mental fluents and actions in addition to physical ones. In $\mathscr{APIA}$, we additionally introduce policy fluents and actions needed to reason over policy compliance (see Table 1). The new policy action descriptions encode the effects of future agent actions on policy compliance and provide means for the control loop to deem non-compliant activities futile and execute compliant ones in their place.

For example, the dynamic causal laws in (5) define inertial policy fluents $auth\_compliance(weak)$ and $auth\_compliance(strong)$ according to the definitions for strong and weak authorization policy compliance of $\mathscr{AOPL}$ seen in Definition 1:

$$\begin{array}[]{lllll}a&\textbf{ causes }&\neg auth\_compliance(strong)&\textbf{ if }&\textbf{ not }permitted(a)\\ a&\textbf{ causes }&\neg auth\_compliance(weak)&\textbf{ if }&\neg permitted(a)\end{array}$$

(5)

These rules are defined for every physical action $a$ of the transition system. Should an action $a$ occur where $permitted(a)$ is not known to be true, then the scenario ceases to be strongly compliant (i.e., it becomes weakly compliant). Since $auth\_compliance(strong)$ cannot be made true again by any action, the rest of the scenario remains weakly compliant by inertia. Likewise, should an action $a$ occur where $permitted(a)$ is false, then the scenario ceases to be weakly compliant (i.e., it becomes non-compliant) and remains in this state by inertia.

For every physical fluent $f$, we introduce a new defined policy fluent $policy\_compliant(f)$. This allows us to reuse the $\mathscr{AIA}$ control loop shown in (1) as is. When the agent controller wants to specify that the agent should achieve goal $f$ in a policy-compliant manner, the controller simply has to initiate action

$$select\_goal(policy\_compliant(f))$$

instead of the original $select\_goal(f)$. The policy fluent $policy\_compliant(f)$ is true iff $f$ is true and $auth\_compliance(l)$ is true, for some minimum compliance threshold $l$ set by the controller. Thus, when $policy\_compliant(f)$ is an agent’s goal, activities below $l$-compliance are deemed as futile and the agent works to achieve fluent $f$ subject $l$-compliance.

One behavior mode is for the agent to strictly adhere to its authorization policy such that it never chooses to perform $ignore\_neg\_permitted(a)$. This causes all non-compliant actions to indirectly cause $policy\_compliant(f)$ to be false, if they are executed. Hence, only activities with weakly or strongly compliant actions are considered. Since this mode never dares to become non-compliant, it is called subordinate.

A similar behavior mode causes the agent to never perform $ignore\_not\_permitted(a)$. This causes all weak and non-compliant actions to indirectly cause $policy\_compliant(f)$ to be false when executed. Hence, only activities with strongly compliant actions are considered. Since weakly compliant actions are actions for which the policy compliance is unknown, this mode is called paranoid as it treats weakly compliant actions as if they were non-compliant.

Another behavior mode allows unrestricted access to the two actions, $ignore\_neg\_permitted(a)$ and $ignore\_not\_permitted(a)$. This mode is called utilitarian because it reduces the behavior of $\mathscr{APIA}$ to that of $\mathscr{AIA}$, where policies are not considered at all.

An interesting feature of the $ignore\_neg\_permitted(a)$ and $ignore\_not\_permitted(a)$ actions is the ability to optimize compliance. Using preference statements in ASP, we can require the control loop to minimize the use of these two policy actions. Hence, if it is possible to execute an activity that is strongly compliant, the agent will prefer it over a weakly or non-compliant one (since the use of these actions is required to allow $policy\_compliant(f)$ to be true). Under this condition, $ignore\_not\_permitted(a)$ and $ignore\_neg\_permitted(a)$ are only used when it is impossible to achieve the fluent $f$ in a strongly or weakly compliant manner, respectively.

The combination of compliance optimization with the first three behavior modes allows for more possible configurations. For example, adding optimization to the subordinate option makes a cautious mode. In this mode, the agent will try to mimic the behavior of the paranoid mode (all strongly compliant actions), but ultimately it will reduce to subordinate (all weakly compliant actions) in the worst case. Likewise, adding optimization to the utilitarian mode adds two options: best effort and subordinate when possible. Best effort prefers strong compliance over weak compliance and weak compliance over non-compliance, but ultimately permits non-compliance when no better alternatives exist. Subordinate when possible prefers weak compliance over non-compliance but does not optimize from weak compliance to strong compliance.

A new feature of this approach to optimization is the ability to optimize within the weakly and non-compliant categories. Consider two weakly compliant activities, 1 and 2, where activity 1 has more weakly compliant actions than activity 2. Since weakly compliant actions do require a concurrent $ignore\_not\_permitted(a)$ action, activity 1 will have more $ignore\_not\_permitted(a)$ actions than activity 2. Hence, activity 2 will be preferred to activity 1, even though they both fall in the weakly compliant category. Gelfond and Lobo [13] do not consider such a feature.

There are five different configurations (or behavior modes) an agent in the $\mathscr{APIA}$ architecture can have regarding its obligation policy (see Table 3). When in subordinate mode, the agent will never use either $ignore\_obl(a)$ and $ignore\_obl(neg(a))$ actions. Hence, all activities achieving $policy\_compliant(f)$ will be compliant with both aspects of its obligation policy. When in best effort mode, the agent prefers using other actions over these policy actions. Hence, activities will be compliant if possible but may include non-compliant elements when no other goal-achieving activities exist. The permit omissions and permit commissions options are variations of these modes. Mode permit omissions is like best effort with regards to $obl(a)$ policy statements, but like subordinate with regards to $obl(\neg a)$ policy statements. Likewise, permit commissions is like subordinate with regards to $obl(a)$ policy statements but like best effort regarding $obl(\neg a)$ statements. Utilitarian mode, like with authorization policies, reduces the behavior of an $\mathscr{APIA}$ agent with respect to its obligation policy to that of an $\mathscr{AIA}$ agent.

For each $\mathscr{APIA}$ configuration, we adjust the definition of $policy\_compliant(f)$ such that excess policy actions are not required. For instance, in the case of an agent with a subordinate authorization mode, we adjust $policy\_compliant(f)$ such that $ignore\_not\_permitted(a)$ is never needed since such an agent always disregards strong compliance.

To begin with a simple case, suppose that two people are in an office space that has four rooms with doors in between them. Room 1 is connected by door $d_{12}$ to Room 2. Room 2 is connected by door $d_{23}$ to Room 3 and so on. Door $d_{34}$ has a lock and is currently in the unlocked position. Suppose our agent, Alice, wants to greet another agent, Bob. This scenario is represented by a dynamic domain description that considers:

• •

  Fluents: $door\_locked(D)$ for each door $D$, $in\_room(P,R)$, $greeted\_by(P,A)$ where person $P$ is greeted by person $A$; and

• •

  Actions: $move\_through(A,D)$, $lock\_door(A,D)$, $unlock\_door(A,D)$, $greet(A,P)$, where $A$ is the person doing the action, $D$ is a door, and $P$ a person (the direct object of the action).

Assume that agent Alice is given a policy specifying that all actions are permitted along with the following pre-computed activity that is stored in her knowledge base as the set of facts:

$$\begin{array}[]{l}\{activity(1),\ \ \ \ length(1,4),\ \ \ \ goal(1,policy\_compliant(greeted\_by(alice,bob))),\\ \ \ component(1,1,move\_through(alice,d_{12})),\ \ component(1,2,move\_through(alice,d_{23})),\\ \ \ component(1,3,move\_through(alice,d_{34})),\ \ component(1,4,greet(alice,bob))\}\\ \end{array}$$

Before the control loop begins, Alice observes that she is in Room 1, Bob is in Room 4, the door $d_{34}$ is unlocked, and that she has not yet greeted Bob.

At timestep 0, the first iteration of the control loop begins. In this first step, Alice analyzes her observations and interprets unexpected observations by assuming undetected exogenous actions occurred. None of her observations are unexpected, so no exogenous actions are assumed to occur. Alice then intends to wait at timestep 0. Alice attempts wait. Alice observes that her wait action was successful and that, in the meantime, the exogenous action

$$select(policy\_compliant(greeted\_by(alice,bob)))$$

happened. The time step is incremented and Alice does not observe any fluents.

The second iteration of the control loop begins. Alice analyzes her observation of
$select(policy\_compliant(greeted\_by(alice,bob)))$ and determines that
$active\_goal(policy\_compliant(greeted\_by(alice,bob)))$ is true. Alice then starts planning to achieve $policy\_compliant(greeted\_by(alice,bob))$ and determines that she intends to start activity 1. Since each action in activity 1 is strongly compliant, no policy actions are needed.

The rest of the example is very straight forward and is almost identical to scenarios discussed by [6, 7] in the $\mathscr{AIA}$ architecture.

Let us consider a less fortunate example, in which a strongly compliant activity becomes weakly compliant due to an unexpected environmental observation. In the same scenario, suppose we modify Alice’s policy from Example A such that regarding $greet(A,P)$ we have:

$$\begin{array}[]{lll}permitted(greet(A,P))&\textbf{ if }&\neg busy\_working(P)\\ permitted(greet(A,P))&\textbf{ if }&busy\_working(P),\ \ in\_room(P,R),\\ &&door\_connects(D,R),\ \ knocked\_on\_door(D)\end{array}$$

(6)

We also have new fluents $busy\_working(P)$ and $knocked\_on\_door(D)$, and actions $begin\_working(A)$ and $knock\_on\_door(A,D)$. Let Alice’s knowledge base contain two additional activities, 2 and 3, with the same goal as 1 and defined by the sets of facts:

$$\begin{array}[]{l}\{activity(2),\ \ \ \ length(2,5),\ \ \ \ goal(2,policy\_compliant(greeted\_by(alice,bob))),\\ \ \ \ \ component(2,1,move\_through(alice,d_{12})),\ \ component(2,2,move\_through(alice,d_{23})),\\ \ \ \ \ component(2,3,knock\_on\_door(alice,d_{34})),\ \ component(2,4,move\_through(alice,d_{34})),\\ \ \ \ \ component(2,5,greet(alice,bob)),\\ \ \ activity(3),\ \ \ \ length(3,2),\ \ \ \ goal(2,policy\_compliant(greeted\_by(alice,bob))),\\ \ \ \ \ component(3,1,knock\_on\_door(alice,d_{34})),\ \ component(3,2,move\_through(alice,d_{34}))\}\end{array}$$