Utility-Based Reinforcement Learning: Unifying Single-objective and Multi-objective Reinforcement Learning

SORL and MORL share a common foundation in the assumption that the environment can be represented as some form of Markov Decision Process (MDP). In the single objective case, a MDP is represented by the tuple $\langle S,A,T,\gamma,\mu,R\rangle$, where:

• •

  $S$ is the state space

• •

  $A$ is the action space

• •

  $T\colon S\times A\times S\to\left[0,1\right]$ is a probabilistic transition function

• •

  $\gamma\in[0,1)$ is a discount factor

• •

  $\mu\colon S\to[0,1]$ is a probability distribution over initial states

• •

  $R\colon S\times A\times S\to\mathbb{R}$ is a scalar-valued reward function

For a single-objective problem of this nature, the agent’s aim is to discover a policy $\pi$ which maximises either the finite-horizon undiscounted reward (Equation 1), the infinite-horizon cumulative discounted reward (Equation 2), or the average reward (Equation 3). We note that the choice of criteria to optimise is a decision made by the system designer, rather than being an inherent property of the MDP itself.

MORL differs from SORL in being based on the concept of a Multi-Objective Markov Decision Process (MOMDP). This is identical to a MDP in all respects other than the reward, where rather than a scalar reward $R\colon S\times A\times S\to\mathbb{R}$, the MOMDP has a vector-valued reward function $\mathbf{R}:S\times A\times S\to\mathbb{R}^{d}$ which specifies the immediate reward for each of the $d$ objectives.

The optimisation criteria in Equations 1–3 are insufficient for a MOMDP, as they only provide a partial-ordering over the vector values. Therefore the utility-based paradigm for MORL assumes the existence of a utility function (sometimes called a scalarisation function) $u$, which maps the multi-objective value of a policy to a scalar value Hayes et al. (2022a). Note that the details of this utility function may or may not be available in advance – in the latter case the agent may need to identify a set of policies which would be optimal under different parameterisations of the utility function. We will denote a parameterised utility function as $u_{\omega}$.

Given the existence of $u$ we can define two alternative optimisation criteria, depending on the stage at which $u$ is applied (the reasoning for this distinction will be discussed further in Section 5)¹¹1Note that these equations relate to the infinite-horizon discounted return which has been the most widely studied criteria in the MORL literature. Similar utility-based equations can readily be defined for the finite-horizon and average-reward criteria.. Equation 4 defines the scalarised expected reward (SER) where $u$ is applied to the expected vector reward, whereas Equation 5 defines the expected scalarised reward, where $u$ is applied inside the expectation operator. In either case, these equations provide a scalar value which defines a total ordering over policies.

In the previous subsection we highlighted the differences between conventional single-objective RL and utility-based MORL. We now consider the commonalities between these approaches, and argue that utility-based RL provides a general framework which encompasses both SORL and MORL²²2The name ‘utility-based is not intended to imply that standard SORL is not maximising a measure of utility, but to emphasise our framework’s explicit consideration of user-utility. In standard SORL the choice of optimisation criteria is made by the developer whereas in multi-policy UBRL the final decision rests with the end-user..

We start with the straightforward observation that MDPs are a subclass of MOMDPs. Any MDP can be mapped to an equivalent MOMDP with a reward function Which consists of a one-dimensional vector $\mathbf{R}=[R]$.

Having done this conversion, then utility-based methods such as those defined in terms of the optimisation criteria in Equations 4 and 5 can be applied to the resultant MOMDP. More generally we might consider any utility function which maps from the expected mean rewards or the expected distribution of rewards to a scalar value – we will explore examples of this in Sections 4.2 and 4.3. In addition the utility function may itself be parameterised, so that a set of policies would be required in order to optimise the criteria across all possible instantiations of $u$.

We note that some authors have previously criticised approaches such as MORL and risk-aware RL as “special cases” Silver et al. (2021), and we anticipate that similar criticisms may be levelled at UBRL. To the contrary, we argue that the UBRL framework presented here is a strict generalisation of prior RL frameworks. It can encompass standard scalar-reward RL by simply setting the number of objectives $n$=1 and the utility function $u$ to the identity function, while also supporting other more complex forms of RL, such as MORL.

Consider the scenario of a mining company which has the choice between carrying out its usual operations or following a riskier course of action that can potentially lead to a lot of the resource being mined (e.g., committing most of the workforce to exploratory excavation). The company has outstanding contracts that commit it to sell a given amount to certain costumers at a set price and leading to an incurred penalty if these amounts are not delivered.

There are various ways such a scenario might be addressed using standard RL approaches:

• •

  We might model this as a constrained RL task Gattami et al. (2021), where the fulfillment of contracts is treated as an inviolable constraint. But perhaps this is overly restrictive, as the company management may be prepared to breach contracts if the potential pay-off is sufficiently high?

• •

  It could be modelled as a SORL problem, with the value of the resource mined represented as a positive scalar reward, and the penalty for unfulfilled contracts as a negative reward. This would allow the agent to select the riskier action if the potential gain offsets the contractual penalty. But such an approach might fail to account for longer-term reputational damage arising from unfulfilled contracts.

• •

  MORL methods could model the reputational harm as an additional objective to be minimised. But this may be very difficult to define quantitatively.

As a result, the preferred approach of the company’s management might be to generate a set of alternative policies using UBRL, and make an a posteriori decision about which policy to follow. This could be accomplished by defining the reward in terms of the quantity of the resource mined, and then specifying the utility function as in Equation 6, where $d$ is the monetary value of each unit of the resource, $c(\cdot)$ is a binary function returning 1 if the contract terms are breached and 0 otherwise, $p$ is the penalty for breaching the contract, and $h$ is an estimate of the financial impact of the reputational harm caused by a contract breach.

Note that the utility term $u$ is conditioned on the value of $h$. Rather than learning an optimal policy for a single estimate of cost of the reputational harm, this utility-based formulation enables the UBRL agent to learn multiple policies, each optimal with respect to a different estimated value of $h$. The multiple policies produced via this process would then be presented to the management, allowing them to make an informed decision about the best policy to execute. Existing multi-policy MORL methods such as the Conditioned Network approach of Abels et al. (2019) could readily be adapted to this task by conditioning Q-values and policies on $h$.

Many authors have investigated risk-aware RL agents – that is, agents which apply some awareness of risk during their decision-making rather than simply aiming to maximise the expected return. A common approach to address this problem is to use a distributional form of RL Bellemare et al. (2023); Hayes et al. (2022b); Nguyen-Tang et al. (2021); Martin et al. (2020), which learns the distribution of future returns from any state rather than just the mean expected return. These approaches can take into account various aspects of this distribution when identifying the optimal policy (e.g. best worst case, more than 60% chance of profit, etc.) Cheng et al. (2023); Greenberg et al. (2022); Yang et al. (2023); Fawzi et al. (2022); Hayes et al. (2023).

Again we would argue that such an approach is essentially already utility-based. In this case the choice of constraints applied to the distribution represents the definition of the utility function. Conventional SORL approaches to risk-awareness assume that these constraints are fixed a priori, and learn a single policy which is optimal under those constraints.

In contrast by adopting the UBRL mindset, we can envisage an algorithm which learns, in parallel using multi-policy methods, a set of policies which are optimal under different risk-sensitivity preferences. These could then be presented to a human decision-maker for selection of the policy to actually be executed.

For example, consider the conditional value at risk (CVaR) defined in Equation 7, which has been widely-used as an optimisation criteria in risk-aware RL Singh et al. (2020). Here $Z$ represents the distribution of future returns, and $\alpha\in[0,1]$ is a parameter controlling the risk-sensitivity of the agent. In conventional risk-aware RL, the value of $\alpha$ is fixed and a single optimal policy is found. If we instead treat CVaR_α as a utility function parameterised by $\alpha$, we can apply multi-policy methods to find policies with a diverse range of different sensitivities to risk, and then allow a human decision-maker to select their preferred policy.

We noted earlier in Section 2 that the choice of optimisation criteria used in SORL (undiscounted sum, discounted sum, or average reward) is made by the system designer/user rather than being an inherent property of the MDP itself. As such, this decision could in itself be regarded as a definition of user utility, although it usually is not framed as such. Similarly we would contend that the discounting term $\gamma$ used in Equation 2 can be viewed as a form of utility definition. This term does not originate from the MDP itself, and two agents using different discounting rates may well derive differing optimal policies from the same MDP. Conventional SORL requires that the discounting rate be fixed prior to learning, and the impact of this choice on the final policy is not evident to the user. The only way to gain insight into the effect of the discounting rate on the agent’s behaviour is to subsequently train another agent on the same MDP using a different value for $\gamma$ which is inefficient.

A superior approach is to treat the discounting rate as a parameter of the utility function as shown in Equation 8, and use a multi-policy UBRL algorithm to simultaneously learn optimal policies for a range of different values for $\gamma$. The resulting policies can then be presented to the user, allowing them to make a fully informed choice of the appropriate policy³³3We note that the utility definition in Equation 8 differs from both forms of utility previously used in MORL (Equations 4 and 5), in that it is the per-time-step reward, rather than the summed return, that is being transformed.

AI safety researchers have argued that ‘hard optimizers’ like conventional RL can be unsafe, as their absolute focus on maximising the given reward signal may lead to adverse side-effects if the rewards are not fully aligned with the actual desired behaviour Taylor (2016); Vamplew et al. (2018). This has lead to interest in developing satisficing agents which are not incentivised to over-optimise Hajiabolhassan and Ortner (2023). For example, in a multi-agent resource-gathering task we might wish to disincentivise an agent from collecting more resources than it actually needs to avoid adversely impacting other agents. In the context of MORL satisficing has been implemented by causing the agent to switch emphasis between objectives once a suitable level of return has been achieved for each objective. This can either use a hard threshold for each objective as in Vamplew et al. (2021) or a non-linear utility which reduces the weighting for objectives as higher returns are achieved Smith et al. (2023).

These approaches are not directly applicable to SORL due to the lack of alternative objectives. For example, reducing the gradient of the utility function as in Smith et al. (2023) would not disincentivise the agent from still trying to maximise it. So for safe SORL we might need to consider the concept of a non-monotonic utility function where the utility actually falls in value after a satisfactory amount of reward has been received, as shown in Equation 9.

This could of course also be achieved via changing the reward definition within the MDP. However that approach would not be amenable to multi-policy learning whereas a multi-policy UBRL algorithm could simultaneously learn optimal policies for various values of the threshold $\omega$, allowing insight into the appropriate value of $\omega$ which will produce the desired safe behaviour while still performing to a satisfactory level in terms of the actual task.