Beyond Preferences in AI Alignment

The rest of this paper is organized as follows: In Section 2, we examine rational choice theory as a descriptive account of human decision-making. Drawing upon the tradition of revealed preferences in economics, rational choice theory is often taken for granted by AI researchers seeking to learn human preferences from behavior. In doing so, they assume that human behavior can be modeled as the (approximate) maximization of expected utility, that human preferences can be represented as utility or reward functions, and that preferences are an adequate representation of human values. We challenge each of these assumptions, offering alternatives that better account for resource-limited human cognition, incommensurable values, and the constructed nature of our preferences.

Developing upon these ideas, in Section 3 we turn to expected utility theory (EUT) as a normative standard of rationality. Even while recognizing that humans often do not comply with this standard, alignment researchers have traditionally assumed that sufficiently advanced AI systems will do so, and hence that solutions to AI alignment must be compatible with EUT. In parallel with recent critiques of this view (Thornley,, 2023, 2024; Bales,, 2023; Petersen,, 2023), we argue that EUT is both unnecessary and insufficient for rational agency, and hence limited as both a design strategy and analytical lens. Instead of adhering to utility theory, we can design tool-like AI systems with locally coherent preferences that are not representable as a utility function. We can also go beyond EUT, building systems that reason about preferences in accordance with deeper normative principles.

After interrogating these descriptive and normative foundations, in Section 4 we consider what this implies for aligning AI with a single human principal. Since reward functions may not capture even a single human’s values, the practice of reward learning is unsuitable beyond narrow tasks and contexts where people are willing to commensurate their values. Furthermore, since preferences are dynamic and contextual, they cannot serve as the alignment target for broadly-scoped AI systems. Rather, alignment with an individual person should be reconceived as alignment with the normative ideal of an assistant. More generally, AI systems should not be aligned with preferences, but with the normative standards appropriate to their social roles and functions (Kasirzadeh and Gabriel,, 2023).

If normative standards are to serve as alignment targets, whose judgments do we consider in determining these (oft-contested) standards? We take up this final topic in Section 5, critiquing naive preference aggregation as an approach to aligning AI with multiple human principals (Fickinger et al.,, 2020). Despite increasing recognition that this approach is inadequate (Critch and Krueger,, 2020; Gabriel,, 2020; Korinek and Balwit,, 2022), applied alignment techniques typically aggregate preferences across multiple individuals, overlooking the contested and plural nature of human values, while conflating norm-specific judgments with all-things-considered preferences. As alternatives, we argue that contractualist and agreement-based approaches can better handle value contestation while respecting the individuality of persons and the plurality of uses we have for AI. This motivates a reframing of the aims of AI alignment as they have often been conceived: Our task is not to align a single powerful AI system with the preferences of humanity writ large, but to align a multiplicity of AI systems with the norms we agree that each system should abide by (Zhi-Xuan,, 2022).

A note on methodology: Whereas most philosophy papers tend to be narrow in scope, this paper is intentionally broad; it covers a wide range of connected topics, and hence makes arguments that are relatively brief. Our aim is not provide a decisive argument for any particular thesis, but to provide a critical review of the role of preferences in AI alignment, while developing a research agenda for alternative approaches that is accessible to an interdisciplinary audience.

The issue with both perfect and noisily-rational models of human decision-making is that they do not account for the systematic deviations from optimality that humans in fact exhibit. As a long line of psychological and behavioral research has shown, humans are boundedly rational at best, exhibiting satisficing instead of optimizing behavior, (Simon,, 1957, 1979). These deviations from optimality include framing effects, loss aversion, anchoring biases, and mis-estimation of high and low probabilities – phenomena which are better modeled by prospect theory (Kahneman and Tversky,, 1979; Tversky and Kahneman,, 1992) than standard rational choice theory. More generally, many of the decision problems that people encounter are computationally intractable to solve optimally, making rational choice a implausible model of human behavior (van Rooij,, 2008; Bossaerts et al.,, 2019; Camara,, 2022). Instead, research suggests that humans make use of a variety of heuristics in order to approximately solve the problems they encounter (Gigerenzer,, 2008).

Challenges to modeling bounded rationality. How might AI systems that infer human preferences and values account for these findings? One approach might be to incorporate a sufficiently long list of known heuristics and biases into our models of human decision-making, thereby ensuring that preferences can be robustly inferred even in the presence of such biases (Evans et al.,, 2016; Chan et al.,, 2021). However, this approach is highly contingent upon on our current state of knowledge about human rationality — what if we miss out important biases in our models, leading to inaccurate predictions and inferences (Christiano, 2015b, ; Steinhardt,, 2017)? As a potential remedy, Shah et al., (2019) suggest learning human biases alongside their preferences. But a conceptual difficulty remains: Without any inductive constraints on the types of errors humans are susceptible to, how can we ensure that human biases are accurately learned? As Armstrong and Mindermann, (2018) show, even inductive preferences for more parsimonious models of human decision-making cannot distinguish intuitively plausible hypotheses from observationally-equivalent but implausible hypotheses, such as the possibility that humans are acting anti-rationally by minimizing the satisfaction of their preferences.

Resource rationality as a unifying frame. To address these challenges, we suggest – in line with prior work – that resource rational analyses of human decision-making might provide an answer: Instead of treating human biases and heuristics as idiosyncratic artifacts, resource rationality posits that seemingly irrational human behavior can often be understood as arising from the rational use of limited computational resources (Lieder and Griffiths,, 2020).³³3Also known as computational rationality (Lewis et al.,, 2014; Gershman et al.,, 2015; Oulasvirta et al.,, 2022), algorithmic rationality (Halpern and Pass,, 2015), and bounded optimality (Russell and Subramanian,, 1994). For example, availability biases towards extreme events can be modeled as a form of resource-rational sampling (Lieder et al.,, 2018), susceptibility to sharing inaccurate information can result from a form of rational inattention (Pennycook et al.,, 2021; Sims,, 2003), and habitual action can be explained as a mechanism for avoiding costly planning under time constraints (Keramati et al.,, 2016). Resource rationality thus serves as a generative principle for hypothesizing possible deviations from standard rationality, and then testing whether such deviations in fact occur in humans.

Resource rationality as an inductive bias. What does this imply for AI alignment? Most practically, the assumption of resource rationality can be embedded as priors over computation time and representational complexity in probabilistic models of human decision-making (Zhi-Xuan et al.,, 2020; Ho and Griffiths,, 2022; Berke et al.,, 2023; Jacob et al.,, 2024), enabling systems to infer human goals and preferences from failed plans and mistaken reasoning (Evans et al.,, 2016; Alanqary et al.,, 2021; Chan et al.,, 2021), while accelerating the speed of goal inference (Zhi-Xuan et al., 2024a, ). Embedding these priors on human resource bounds provides a strong but flexible inductive bias on the the space of decision procedures that humans might employ. Unlike a simplicity prior, this may avoid concerns about the non-identifiability of human preferences (Armstrong and Mindermann,, 2018).

The normative appeal of resource rationality. Indeed, the inductive bias imposed by resource rationality has a normative appeal over a simplicity-based approach: It tries to make sense of humans as rational creatures, aiming for teleological explanations of our behavior instead of reducing us to mere physical phenomena to be explained by the simplest causal mechanism. At the same time, it is a forgiving standard of rationality, allowing room for mistakes when inferring preferences from their decisions, while placing greater evidential weight on decisions made after lengthier deliberation. Both of these features make resource rationality a promising framework for systems that learn our values: Rather than directly associating our behavior with our preferences, preferences are associated with how we would act if we were more thoughtful, reflective, and informed.

While resource rationality provides a more flexible framework for modeling the relationship between preferences and behavior, this says little about how preferences themselves should be represented. For the most part, resource rational analyses continue to represent human preferences in terms of scalar costs and rewards, or more generally, utility functions, with the primary innovation being the inclusion of costs on computation (Lieder and Griffiths,, 2020; Callaway et al.,, 2022). Yet, there are many reasons to think that reward functions and utility functions are inadequate representations of human preferences, while also tending to produce conceptual confusion about what they do represent.

The limited expressivity of reward functions. These issues are most easily appreciated in the case of (scalar, Markovian) reward functions. As noted earlier, the reward representation assumes that the utility of a sequence of states and actions $\xi=(s_{1},a_{1},...,s_{T},a_{T})$ can be decomposed into a sum of scalar rewards over time:

$$U(\xi)=\textstyle\sum_{t=1}^{T}R(s_{t},a_{t})$$

Advocates of the reward representation argue that any task accomplishable by an intelligent agent can be framed as a reward maximization problem (Silver et al.,, 2021). As Kasenberg et al., (2018) point out, however, this minimally requires that all historically relevant information is already included in the representation of each state $s_{t}$ — a requirement since stated more formally by Abel et al., (2021) and Bowling et al., (2023). This means that without careful feature engineering, reward functions cannot easily express time-extended preferences like the desire to keep a promise, or the value of narrative coherence. Separately, the scalar nature of the (standard) reward representation means that it cannot represent the existence of incomplete preferences due to multiple incommensurable scales of value (Vamplew et al.,, 2022; Anderson,, 1995; Chang,, 1997): Sometimes, the choices before us may seem good or bad in such distinct ways that it makes no sense to say which is better than another.⁴⁴4For example, one might have to choose between staying in a democratic country while being at severe risk of poverty, or immigrating to a country with material security but no political freedoms. As a result, we may have preferential gaps: pairs of options where neither option is preferred over the other, nor are they equally preferred.

Confusion about what reward functions represent. Alongside these limitations in expressiveness, there is often slippage among AI researchers regarding the ontological status of reward,⁵⁵5See Lambert et al., (2023) for an overview in the context of reinforcement learning from human feedback. which is sometimes interpreted as the intrinsic desirability of a particular state or action (Schroeder,, 2004), or as a biological signal that promotes learning (Butlin,, 2021) or evolutionary success (Singh et al.,, 2009), but is also used to define the instrumental value of a state (as in reward shaping (Ng et al.,, 1999; Booth et al.,, 2023)), or to demarcate goals (i.e. desired trajectories or states of affairs (Molinaro and Collins,, 2023; Davidson et al.,, 2024)). While this is partly a testament to the flexibility of reward functions as a mathematical formalism, this also means that distinct normative concepts (preferences, goals, intents, desires, values, etc.) get conflated or subsumed under the label of “reward”. In alignment research, this manifests as the tendency to frame value alignment in terms of reward learning (Hadfield-Menell et al.,, 2016; Leike et al.,, 2018), and to formalize concepts like “goals” (Di Langosco et al.,, 2022) and “intents” (Ouyang et al.,, 2022) as reward functions. This is despite the existence of other useful and potentially more appropriate formalisms, such as the formalization of goals as logical specifications (Fikes and Nilsson,, 1971), and the formalization of intentions as (partial) plans (Bratman,, 1987; Bratman et al.,, 1988).

Utility functions are more expressive, but insufficiently constrained. While not without their own interpretive confusions,⁶⁶6Most prominently, the debate between interpreting utility as cardinal measure of welfare that is comparable across individuals, versus a mere representation of individual preference rankings (Strotz,, 1953; Harsanyi,, 1953) utility functions are considerably more general than (Markovian) reward functions. For example, they can be defined over arbitrarily long sequences of states, allowing them to capture time-extended preferences. However, what utility functions buy in terms of expressiveness comes at a cost to both identifiability and tractability: If no constraints are placed on the structure of human utility functions, then given some sequence of actions (e.g. a person buying ten apples, then two oranges), it is not possible to disambiguate a reasonable utility function that explains the actions (e.g. by assigning higher utility to an apple over an orange) from a degenerate utility function that assigns a utility of one to exactly the observed sequence.⁷⁷7See Armstrong and Mindermann, (2018) for a similar argument. Note that these identifiability problems already exist with Markovian reward functions (Cao et al.,, 2021; Kim et al.,, 2021; Skalse et al.,, 2023), but are made worse once we let go of the Markov assumption altogether. In addition, many utility functions are intractable to coherently maximize (Camara,, 2022) or even to compute.⁸⁸8For example, a utility function might embed the NP-hard traveling salesperson problem (TSP), by assigning higher utility to road networks with TSP solutions under a certain cost threshold. While a human could hold such preferences, it would generally be very costly for them to check whether those preferences hold. If we apply the principle of resource rationality here too, this makes intractable utility functions less plausible representations of human preferences. Finally, utility functions are not without their own expressivity limitations: Like scalar rewards, they assume away preference incompleteness due to plural and incommensurable values (Chang,, 2021; Eckersley,, 2018). Indeed, empirical work shows that incomplete preferences are not just possible, but actual (Cettolin and Riedl,, 2019; Nielsen and Rigotti,, 2023). This means that utility functions are, at best, approximate representations of human preferences, not exact ones.

Fundamental tensions for any representation of preferences. It is worth noting that these tensions between expressivity, structure, and tractability apply to any representation of human preferences, not just reward or utility functions. Thus, while it might be tempting to ensure expressivity by directly representing human preferences as a (possibly incomplete) list of comparisons over universe trajectories (or a distribution over such comparisons (Dumoulin et al.,, 2024)), such a list would be extremely space-inefficient, while providing little to no action guidance in novel choice situations. Instead, we should recognize that part of what makes reward and utility functions so useful in practice is that they are typically engineered to be compact representations of preferences. Practically useful alternatives should maintain this property, while better capturing the richness of human preferences.

Alternative representations can better capture temporal structure and value plurality. Fortunately, many promising options exist: Temporal logics (Kasenberg et al.,, 2018) and reward machines (Icarte et al.,, 2022; Davidson et al.,, 2024) avoid the limitations of traditional reward functions, enabling the expression of time-extended preferences. At the same time, they can be structured in a way that enables effective learning from human behavior (Shah et al.,, 2018; Zhou and Li,, 2022). To account for incommensurability and incompleteness, vector-valued reward functions (Vamplew et al.,, 2022), conditional preference networks (Boutilier et al.,, 2004; Cornelio et al.,, 2013), or interval-valued utility functions (Denoeux and Shenoy,, 2020) can be used, allowing our models to explicitly surface hard choices due to preferential gaps. Many of these representations are also associated with rich compositional semantics, making apparent the complex internal structure of human goals and preferences (Gerevini and Long,, 2005; Davidson et al.,, 2024). Although these formalisms have limitations of their own, they nonetheless embed important insights about how preferences can be computationally represented. As such, they deserve further study by alignment researchers seeking to adequately model human preferences in a general fashion, while also being useful representational tools for today’s AI systems.

Preferences are constructed, not basic. Thus far, we have proceeded as if human motivations and values are adequately captured by the concept of “preference” as it is used in rational choice theory. But as far as evaluative concepts go, this concept of “preference” is an extremely thin one: Mathematically, a “preference” is just some ordering of two options, which can be interpreted as either a disposition to choose one option over another, subjective liking of one option over the other (Franklin et al.,, 2022), or an all-things-considered judgment in favor of one of the options. Distinct as these interpretations are, what they share is their highly abstract and general nature — “preference” is a thin concept because it does not encode richer semantic information beyond the bare notion of “betterness”. Insofar as utility functions are interpreted as representations of preferences, this thinness is inherited by them: Utility just represents the mere preferability of some option. But why exactly are some options preferred over others? In virtue of what reasons do people make these preference judgments? Without answering these questions, we are unlikely to model how someone’s preferences generalize to novel options in ways they would endorse. To do so, we must go beyond preferences as the fundamental unit of analysis, and understand how preferences are computed and constructed from our reasons and values (Warren et al.,, 2011; Lichtenstein and Slovic,, 2006).

Rational choice as action on the basis of reasons. In making this point, we depart from the domain of rational choice theory, and return to a more basic understanding of what it means to model ourselves as rational agents: We are agents that take ourselves to act on the basis of reasons (Raz,, 1999; Logins,, 2022).⁹⁹9While some psychological theories deny that reasons are the causes or motivations for human action (at least typically), they can nonetheless serve as justifications for our actions (Mercier and Sperber,, 2011, 2017). As such, insofar as our goal is to build AI systems that infer justified bases of action from our behavior (and then act according to them), reasons can still play this role. These reasons might include desires, such as an intrinsic desire to avoid pain (Sinhababu,, 2017), evaluative judgments, such as the judgment that a movie is artistic enough to be worth watching (Anderson,, 1995), or even acts of will, such as the intention to pursue a specific career (Chang,, 2009).

Evaluative concepts as building blocks for reasons. What exactly is the content of these reasons? In decision theory and Humean accounts of motivation (Sinhababu,, 2017), only beliefs (represented as subjective probabilities) and desires (represented as the utility of some desired outcome) are considered as reasons for action. But even if we set aside other accounts (Anderson,, 1995; Chang,, 2004; Parfit,, 2018), this leaves open what a person’s beliefs and desires are about. If I desire to be both helpful and honest to others, what does it mean to be helpful or honest? Acting upon this desire requires applying the concepts of helpfulness and honesty, which are not just any concepts, but evaluative concepts, or values. Importantly, most such concepts are not thin ones, like preference, utility or goodness; they are thick evaluative concepts — concepts that comprise both descriptive and normative elements — such as beauty, humor, or health. As Blili-Hamelin and Hancox-Li, (2023) point out, even the concept of intelligence so central to AI is thick in this way.

Utility functions as aggregators of distinct evaluative judgments. How should AI systems model such evaluative concepts, and their relationship to preferences and action? As a first pass, one might turn the utility representation theorems on their head, viewing reward and utility functions as generators of human preferences, instead of mere representations of them. Indeed, as gestured at earlier, reward and utility functions are often interpreted in this way, with rewards, costs, and utilities respectively treated as biological signals (Singh et al.,, 2009), energetic expenditure (Ab Azar et al.,, 2020), or units of pleasure (Bentham,, 1789). Preferences can then be treated as downstream comparisons of these more basic quantities, as assumed in reinforcement learning from human feedback (Christiano et al.,, 2017; Knox et al., 2024b, ; Zhu et al.,, 2023). Taking this line of thought further, one might treat evaluative concepts such as “aesthetic quality” or “helpfulness” as features over which a reward or utility function is defined, reducing the problem of “value learning” to one of representation or feature learning (Barreto et al.,, 2017; Bobu et al.,, 2022, 2024). On this interpretation, reward and utility functions represent aggregate evaluative judgments, with each feature corresponding to a distinct way of valuing the world.

Utility functions assume that values are always commensurable. Although there is much to be said in favor of this approach, we believe that it is not quite enough. For one, it is still subject to the representational limits of reward and utility functions. In particular, if utility functions are used to represent aggregate value judgments, this effectively assumes that distinct human values are always commensurable in some way, and that our resulting preferences are always complete. Yet, as value pluralists argue, there are contexts where it seems hard or impossible to commensurate our values (Anderson,, 1995), resulting in choices where our reasons run short, and we cannot say if one option is ultimately better than another (Chang,, 1997).¹⁰¹⁰10See our immigration example from earlier, where it may be unclear how to prioritize between political freedom and material security when deciding whether to migrate. Even when we do commensurate our values, utility functions do not provide further information on our reasons and justifications for those trade-offs.

Evaluative judgments are not reducible to observable features. For another, by conceiving of evaluative concepts as “features”, we risk over-simplifying the semantics of many evaluative domains. Consider, for example, the concept of whether a research paper is novel, or whether an action is helpful or universalizable. Applying these concepts requires a complex set of computations: novelty involves evaluating the contributions of a paper with respect to a broader field of established knowledge (Amplayo et al.,, 2019); helpfulness involves estimating the goals of the agent being helped, and then judging whether the action aided in achieving that goal (Ullman et al.,, 2009); universalizability involves simulating what would happen if everyone took a particular action (Levine et al.,, 2020; Kwon et al., 2023b, ). The structured nature of these concepts suggests the need for a suitably rich language of thought — one that captures the compositionality and algorithmic complexity of human conceptual cognition (Piantadosi and Jacobs,, 2016; Quilty-Dunn et al.,, 2023; Wong et al.,, 2023).

Explicitly modeling processes of evaluation and commensuration. To begin to capture all of this complexity, we propose that human decisions can be productively modeled as a three-stage process: Evaluate, Commensurate, then Decide (ECD).¹¹¹¹11Note that this a descriptive framework for modeling how human reasons and values lead to decisions, not a prescriptive framework for designing AI systems. We take up the latter topic in Section 2. Given some choice options, a set of evaluation procedures compute valuations or rankings of the options under consideration, where each procedure corresponds to a distinct evaluative concept. These valuations serve as inputs to a commensuration procedure (Espeland and Stevens,, 1998), which produces, where possible, a context-sensitive value assignment or preference ordering over the options (optionally with justifications for why certain trade-offs were made), while leaving certain preferences unspecified when some options are not comparable. Finally, a decision procedure computes actions and policies with respect to the (possibly incomplete) preference ordering induced by the evaluation and commensuration procedures, resulting in behavior that approximately satisfies those preferences.¹²¹²12One possible instantiation of this framework is multi-objective reinforcement learning (Vamplew et al.,, 2022): Each component of a vector-valued reward function can be thought of as a separate evaluation procedure. These can be transformed by the commensuration procedure into a lexicographic ordering (where some dimensions of value matter infinitely more than others) or constrained maximization problem (where some values must stay within a certain range while others are maximized). A planning or learning algorithm then serves as the decision procedure, producing an action policy that satisfies the commensurated preferences. By explicitly modeling human decisions in this way, we can maintain the distinctness of the values that guide our actions, while foregrounding the ways in which we commensurate our values and dynamically construct our preferences.¹³¹³13In proposing this framework, we do not mean to imply that humans are always going through these stages for every decision; as suggested by the RL formalism, one or more of these procedures may be cached through experience and learning, enabling habitual action without explicitly representing values in the brain (Keramati et al.,, 2016; Hayden and Niv,, 2021). Nonetheless, we can still rationalize learned behavior and cached preferences in light of someone’s values.

Learning and specifying evaluative concepts. This still leaves open the question of how evaluative concepts can be specified or learned. In principle, an AI system could infer such concepts from human decisions by inverting the ECD process, extending inverse reinforcement learning (Ziebart et al.,, 2008) and Bayesian inverse planning (Baker et al.,, 2009). However, decisions alone might provide insufficient information about the nature and structure of our evaluative concepts. Recent advances in large language models (LLMs) suggest a promising alternative: By imitating the distribution of human text, LLMs appear to learn the conceptual roles associated with particular words (Piantadosi and Hill,, 2022), and recognize semantic entailments between sentences (Merrill et al.,, 2024). Correspondingly, they might approximate the semantics of many evaluative concepts (Leshinskaya et al.,, 2023). This may explain why LLMs can often use evaluative adjectives in their appropriate contexts (Mahowald,, 2023), and even perform rudimentary forms of moral reasoning (Jin et al.,, 2022). Still, LLMs remain limited in their ability to represent and reason with compositional concepts (Dziri et al.,, 2023; Mahowald et al.,, 2024; Ramesh et al.,, 2024), and would function as poor models of humans on their own. Instead, we could embed their approximate semantic knowledge into more structured models of human cognition (Kwon et al., 2023a, ; Wong et al.,, 2023) such as the ECD process described above. In doing so we might eventually model the full richness of human practical reasoning.

Coherence is not rationally required. However, coherence arguments for expected utility theory are not as strong as the AI alignment literature has often presumed. The most extensive version of these arguments is given by Gustafsson, (2022), who provides a money pump argument for preference completeness, and then uses completeness to derive arguments for transitivity, continuity, and independence. Yet, as Thornley, (2023) points out, the argument for completeness depends on particular assumptions about how agents are permitted to choose when offered a series of potentially exploitative trades, which can be avoided as long as agents do not accept offers that are less preferred than options they previously turned down.¹⁵¹⁵15Note that whereas Gustafsson, (2022) is focused on justifying the VNM axioms as requirements of rationality (in part by introducing and arguing for other principles of rationality, such as Decision-Tree Separability), Thornley, (2023) is focused on whether the VNM axioms will apply to advanced AI systems, and takes no position on whether they are rationally required. Here we go one step further, and suggest that arguments by Thornley, (2023) and Petersen, (2023) place strong pressure on Gustafsson’s acceptance of rationality principles like Decision-Tree Separability, and hence the argument that the VNM axioms are rationally required. Petersen, (2023) formalizes this counter-argument further, proposing a dynamic choice rule that ensures agents with incomplete preferences are invulnerable to money pumps.¹⁶¹⁶16Analogous arguments have made in defense of imprecise probabilities (Bradley and Steele,, 2014), since they imply incomplete preferences. See also Laibson and Yariv, (2007) on how non-EU preferences are protected by competitive markets, and von Widekind, (2008) on how non-EU preferences can be evolutionarily stable. Indeed, it is accepted by many decision theorists that preference completeness is not a requirement of rationality; instead, all that is required is for an agent’s preferences to be coherently extendible (Steele and Stefánsson,, 2020). In turn, this implies that rational agents need not be representable as EU maximizers.

Coherent EU maximization is intractable. But let us imagine that coherence arguments do go through after all. Even if this were the case, it is far from obvious that advanced intelligences would comply with the axioms of utility theory (or be incentivized to do so) in the face of computational and practical limitations. As Bales, (2023) argues, behaving as an expected utility maximizer can come with considerable costs, while only providing limited benefits. In fact, as we noted in Section 2, most utility functions are computationally intractable to coherently maximize: Camara, (2022) shows that while certain simple classes of utility functions allow for rational choice behavior to be computed in polynomial time, for a large class of other utility functions, agents cannot tractably compute choice behavior that complies with the rationality axioms, and must instead resort to approximately maximizing their utility function. Alternatively, agents may insist on complying with the rationality axioms, but give up on even approximate optimality with respect to their original utility functions. In other words, it is not always resource rational to maximize expected utility.

Coherence alone is not informative. Suppose we could set aside these tractability worries as well.¹⁷¹⁷17Perhaps because it is proven that P = NP, or because advanced AI systems will have such vast resources at their disposal that all relevant intractable problems will be solvable in practice. Even so, it is unclear what information EUT provides us. As discussed by Shah, (2018), Ngo, (2019), and Bales, (2023), many kinds of behavior can trivially be described in terms of utility maximization, including an “agent” that does nothing at all. This means that EUT alone does not say much about the kinds of goals that advanced AI systems are likely to pursue, or what they are likely to do in order to pursue them. While it is possible to draw some conclusions about utility maximizing agents (Soares et al.,, 2015; Turner et al.,, 2021; Everitt et al.,, 2021; Carroll et al.,, 2023), further assumptions are typically needed (e.g. constraints on the space of utility functions) before one can obtain stronger analytical results. Moreover, many deployed AI systems cannot be fully analyzed by EUT, as they are highly approximate (e.g. deep reinforcement learning agents).

Alternative analytical lenses to EUT. What alternatives might one turn to instead to ground understanding, prediction, and alignment of advanced AI systems? Since many others have already addressed some version of these questions, we offer here a brief taxonomy of approaches.

Mechanistic analyses. The most common of such approaches are mechanistic analyses, which reason about the likely properties of AI systems by assuming specific classes of training processes or algorithmic procedures. For example, reasoning about the training dynamics of deep (reinforcement) learning systems can suggest pathways to power-seeking or deceptive behavior (Ngo et al.,, 2022; Di Langosco et al.,, 2022; Krakovna and Kramar,, 2023), or give us confidence that deceptive alignment is unlikely (Wheaton,, 2023). Similarly, knowledge of the workings of general-purpose algorithms, such as model-based search techniques or approximate Bayesian inference methods, can deliver us predictions or even provable guarantees regarding the risk or safety of an AI system (Yudkowsky,, 2015; Bengio,, 2023; Dalrymple et al.,, 2024).

Economic and evolutionary analyses. One downside of mechanistic analyses is that they are tied to particular hypotheses about how AI systems are likely to be built. Given uncertainty about which AI paradigms will ultimately reign dominant, we might want to abstract away from the details of any particular class of AI architectures. While this was the original appeal of EUT analyses, other approaches may hold more promise: economic analyses and evolutionary analyses can respectively ground predictions about the behavior and capabilities of AI systems in what is likely to be economically competitive, or what is likely to be evolutionary successful. For example, economic incentives could imply that AI services are more likely to proliferate than AI agents (Drexler,, 2019), while evolutionary arguments can help us reason about whether increasingly capable AI systems are likely to displace human control over the economy (Hendrycks,, 2023).

Resource-rational analyses. Finally, it may be possible to analyze AI systems through the lens of computational tractability and resource rationality, applying ideas from the study of human cognition to understanding the potential capabilities and limits of artificial cognition (van Rooij,, 2008; Lieder and Griffiths,, 2020). For instance, AI safety via debate can theoretically solve PSPACE problems if optimal play is assumed¹⁸¹⁸18Note that achieving optimal play, formalized as finding a Nash equilibrium, is itself computationally intractable for most games. (Irving et al.,, 2018), while Zhi-Xuan, (2022) cites intractability as a reason to avoid centralized AI planners as an alignment solution, and Van Rooij et al., (2024) provide an intractability argument against the possibility of human-like AI via imitation learning. By and large, however, resource rational analyses of AI systems appear to be neglected. It is thus a potentially fruitful avenue for better analyzing future AI systems — one which retains many of the appealing features of expected utility theory, but adopts a more feasible normative standard.

If agents are neither rationally required nor practically required to act as if they are expected utility maximizers, this opens up the design space of (advanced) AI systems that we might hope to build and align. In particular, we have the option of building AI systems that do not comply with one or more of the axioms of expected utility theory — systems that are not globally coherent in the way that expected utility maximizers are required to be.

Non-globally coherent AI may be more faithfully and safely aligned. Why might this be desirable? There are two broad reasons. One reason is faithfulness. As we discussed in Section 2, many human preferences may be incomplete due to incommensurable values, and we might want AI systems to faithfully represent that preferential structure when making decisions (Eckersley,, 2018). Otherwise, such systems might reliably take actions that promote certain outcomes over others, even though we have yet to form a preference over which of those outcomes is better.¹⁹¹⁹19For example, AI systems that influence or manipulate humans into choosing particular career paths or societal structures because they are programmed to regard them as the best options, instead of respecting our initially incomplete preferences over careers or societal structures. Another reason is safety — for a wide range of (time unbounded) utility functions, expected utility maximizers have been shown to seek power over their environment (Turner et al.,, 2021), and avoid being shut down by their creators (Soares et al.,, 2015),²⁰²⁰20Provided that such utility maximizers are aware of the existence of a shutdown button. suggesting that sufficiently capable utility maximizers will create considerable risks if their utility functions are not compatible with human safety (Carlsmith,, 2022).

AI tools as locally coherent agents. A general class of AI systems that seem to largely satisfy faithfulness and safety are what we might intuitively think of as tools. We use tools to perform tasks that are context-specific — the goals we use them for vary by context — as well as local — we do not expect or want them to reliably affect the world beyond the contexts of their use. Insofar as these tools can be thought of as agents, they are at best locally coherent ones. In this sense, they mimic the role-specific nature of human preferences. Just as people have differing goals and obligations depending on whether they are in the role of a parent or a worker (Anderson,, 1995), tools take on the aims and constraints of their users, whether those involve classifying images or generating code. Within each context, we are typically willing to commensurate our values such that our preferences can be represented as a local utility function, even if we are unwilling to do so in general.

Tool-like locality through local scope. How can we build AI systems that function as tools? The answer, of course, is that we already have: Most AI systems that exist today are best thought of as tools. This is not due to any special care on our part as designers, but only because functioning as a tool is the default nature of rule-bound, computationally limited algorithms with no representation of their own existence in the world. Such algorithms execute a bounded amount of computation in response to some input, terminating when they find an answer or if time runs out. They exhibit no preference for altering the conditions of their termination, or for gaining control over more of their environment, because they cannot even represent the environment they exist in. In other words, such systems are local in scope. This is the case even for systems that we might be tempted to call agents due to their long horizon reasoning abilities (e.g. classical planners, theorem provers) or relative autonomy (e.g. self-driving cars, robot vacuums). To the extent that such systems can be represented as utility maximizers, they can often be viewed as having local, time-bounded utility functions, which provide no incentive for continued operation beyond a certain time or resource bound (Dalrymple,, 2022). Very plausibly, we could even build highly advanced, economically transformative AI systems by composing these bounded tools (Drexler,, 2022; Dalrymple,, 2024).

Maintaining locality despite global scope. Suppose, however, that some actors want to build advanced AI systems that are not bounded in these ways. For example, many AI companies are keen to develop general purpose AI assistants, which follow human instructions in a wide range of domains and contexts, remain operational across contexts, and possess enough understanding of the wider world that they can represent both themselves and their users as entities in that world model. LLMs are increasingly being used in this way, and while their reasoning capabilities remain unreliable and limited (Valmeekam et al., 2023b, ; Dziri et al.,, 2023; Momennejad et al.,, 2024), one might imagine augmenting or embedding them within systems with more coherent representations and reasoning abilities (Parisi et al.,, 2022; Sumers et al.,, 2024). Can we ensure that such systems continue to function as tools, despite their increasingly global scope?

Contextual reward functions are insufficient for locality. We suggest that the answer may depend on whether such systems remain local in terms of the completeness of their preferences, despite having global scope. What does it mean for preferences to be only locally complete? Consider one tempting but unsuccessful way to formalize this idea: We design our system to have a context-sensitive reward function $R(s,c)$, where $s$ is the current state, and $c$ is the current context (e.g. an instruction or prompt given to a LLM-based assistant). The hope is that users will be able to set $c$ to whatever they like, and the system will change the task it optimizes for. Within the context $c$, the system exhibits locally coherent behavior, since its preferences are given by the reward function $R(\cdot,c)$. However, since our system has global scope, it also cares about rewards across contexts: its utility function for a trajectory $\xi=((s_{1},c_{1}),...,(s_{T},c_{T}))$ is $U(\xi)=\sum_{t=1}^{T}R(s_{t},c_{t})$. This means that the system will have a context manipulation incentive, i.e. an incentive to enter and remain within contexts that deliver more reward. For example, it might persuade or manipulate the user to give it instructions that are easier to satisfy.²¹²¹21This can be viewed as a generalization of the shutdown problem (Soares et al.,, 2015): Shutdown implies switching from a context that delivers some reward to a context which never delivers reward. The reason for this is that the system’s preferences are still globally complete — they are represented by a global utility function, despite being context-sensitive.

Tool-like locality through local completeness. How could locally complete preferences avoid these context-manipulating incentives? Following recent work by Thornley, (2024) on circumventing the shutdown problem via incomplete preferences, we formulate local preference completeness as follows: Within each class of trajectories with a fixed schedule of $k$ contexts $(c_{1},...,c_{k})$ that take effect at times $(1,t_{1},...,t_{k-1})$, there is a complete preference ordering over trajectories. Across these classes, trajectories are incomparable, leading to preferential gaps ²²²²22This construction builds upon the incomplete preference condition described in Thornley, (2024) for building agents that are neither shutdown-seeking nor avoiding.. Agents with such preferences would still optimize their behavior while within each context. At the same time, they would exhibit no reliable disposition towards being in some contexts more than others, or manipulating the schedule of contexts. At least in the sense we identified earlier, they would function as tools.

In making this proposal, we do not mean to imply that it is impossible to align or ensure the safety of globally coherent agents — it may be possible to avoid pathological incentives by maintaining uncertainty over the utility function to maximize (Hadfield-Menell et al.,, 2016; Hadfield-Menell et al., 2017a, ), or by carefully balancing utilities across contexts (Armstrong and O’Rourke,, 2017; Holtman,, 2019). We also do not claim that incompleteness is necessary for tool-like AI — if we coordinate to ensure that powerful AI systems always remain bounded and local in scope, then we may never need to explicitly engineer incompleteness. Indeed, it remains unclear how to perform such engineering at scale.²³²³23Thornley et al., (2024) describes a reinforcement learning scheme, but it may not apply to context switching. Nevertheless, if we want to build AI systems that safely respect our preferences and values, it makes sense to keep our options open, and look beyond the default theoretical assumption of globally coherent agents.

EUT does not explain when our preferences are normatively acceptable. Up to this point, we have primarily critiqued the normativity of expected utility theory on formal grounds, drawing upon arguments from decision theory and computational complexity theory. But an arguably deeper problem with EUT is that it fails to ground the normativity of our preferences. EUT is a theory of instrumental rationality not value rationality:²⁴²⁴24A distinction introduced by Weber, (1978). It tells us how to choose our actions in order to satisfy our preferences, and imposes constraints on what those preferences can be, but it does not say anything further about where those preferences can or should come from. Yet, as we have elaborated in Section 2, human preferences are not fundamental, but derivative — they derive from our values and reasons. EUT is thus woefully incomplete. It might tell us how to derive instrumental preferences from intrinsic ones,²⁵²⁵25In the sense that the expected utility of some state or action can be derived from the expected utility of the states it allows us to achieve. but it provides no guidance on many questions of great normative importance, such as why and how to value human and animal lives, whether and when it is permissible to give up equality for efficiency in a democracy, or how to judge the desirability and relevance of EUT itself.

Normative judgments are increasingly automated. Reasoning about these normative questions has traditionally been the purview of humans alone. Indeed, there are many reasons to preserve that state of affairs, lest we cede our moral and political autonomy entirely to machines (van Wynsberghe and Robbins,, 2019). But even without replacing human autonomy over normative affairs, we are already building AI systems that automate normative judgments, assist us with normative reasoning, or operate under normative uncertainty. For example, machine learning methods are routinely used to moderate content that may be regarded as toxic and offensive (Gorwa et al.,, 2020), or to steer LLMs towards producing outputs that are less harmful (Bai et al., 2022a, ). More ambitiously, AI writing assistants are being used to draft legal arguments by mimicking certain aspects of legal reasoning (Iu and Wong,, 2023; Lohr,, 2023). If these trends continue, then increasing amounts of work will have to be done to ensure that AI systems produce normatively appropriate behavior. Humans will either have to do work upfront — a difficult task, given the combinatorially large space of situations that increasingly autonomous systems might encounter — or we will have to imbue AI systems with some semblance of normative reasoning.

The need for theories of normative reasoning. What options do we have for doing this? What would it look like to reason about the preferences and values one ought to have? Given the complexity of these questions, one might hope to sidestep the need for a formal account like EUT entirely, and instead train AI systems to imitate human normative reasoning. This is exemplified by the standard training objective of LLMs, which incentivizes replication of human-generated text. By training such systems on normative human judgments, one might hope that LLMs will learn the reasoning patterns that produce such judgments (Jiang et al.,, 2021). Recent methods such as Constitutional AI (Bai et al., 2022b, ) take this idea one step further, bootstrapping an LLM’s ability to approximate human normative judgments by generating self-critiques (Saunders et al.,, 2022) and revisions, then finetuning the LLM on its own revisions. However, even strong LLMs currently struggle to reproduce human judgments on sufficiently nuanced normative questions (Jin et al.,, 2022; Kwon et al., 2023a, ), and there are reasons to doubt whether LLMs can learn to reliably reason through either imitation (Van Rooij et al.,, 2024; Dziri et al.,, 2023) or self-critique (Stechly et al.,, 2023; Valmeekam et al., 2023a, ). This unreliability suggests that we might want formal theories of normative reasoning after all. Without such theories, we would have no general way of evaluating whether an AI system reasons “correctly”, beyond comparison to often fallible human judgments.²⁶²⁶26While formal theories of reasoning will ultimately have to be evaluated against human judgments themselves, they deliver systematicity and precision that many AI systems do not. Just as with mathematics, logic, and probability theory, formal reasoning systems can succinctly express what we would reflectively endorse, provided that we accept certain principles of reasoning as sound. Perhaps imitation or self-critique will be enough for the majority of everyday situations, but if we want AI systems to address normative questions that are increasingly far afield from past human experience, the ability to validate or produce long chains of normative reasoning may be crucial for both system evaluation and scalable oversight.

Computational theories of normative reasoning. Thankfully, alignment researchers do not have to develop theories of normative reasoning from scratch. Across philosophy, AI, and legal computing, there have been numerous attempts to formalize the logic of argumentation, preferences, and duties, providing systems for reasoning about what we ought to endorse, prefer, or act upon. Abstract argumentation frameworks can be used to compute sets of acceptable arguments given a system of attack relations (Dung,, 1995). Preference logics can be used to express and deduce preferences for some propositions over others (von Wright,, 1972; Liu,, 2011). Deontic logics can be used reason about what norms must be complied with, and which norms are entailed by others (von Wright,, 1951). Many extensions and combinations exist, including argumentation frameworks that allow for reasoning over preferences (Amgoud and Cayrol,, 1998; Modgil,, 2009), or reformulations of deontic logic using preference logic (Hansson,, 1990; Liu,, 2011). Uncertainty over normative arguments and conclusions can also be handled through weighted argumentation frameworks (Amgoud et al.,, 2017) and probabilistic logics (Ng and Subrahmanian,, 1992; De Raedt and Kersting,, 2003), allowing us to avoid over-extrapolation of our normative judgments and dogmatism about “normative truths”. For the purposes of AI alignment, the work that remains to be done is not so much one of formalization, but integration: How can these reasoning systems interface with or augment the standard formalisms of probability theory and decision theory? And how can they be combined with algorithms for machine learning and decision-making?

Integrating normative reasoning with machine learning. One relatively straightforward path to integration might be to use normative reasoning frameworks as synthetic data generators: Instead of directly training machine learning systems on human normative judgments, algorithms for normative reasoning could be used to produce sets of internally consistent arguments that can be derived from an initial set of human-provided judgments. Similar to deductive closure training for classical logic (Akyürek et al.,, 2024), machine learning systems (e.g. LLMs) could then be trained on the sets of derived judgments and arguments,²⁷²⁷27Note there might be multiple sets of valid or defensible arguments, since an initial set of normative premises might be in conflict without decisively ruling each other out (Dung,, 1995) Maintaining this multiplicity may be crucial to avoid normative dogmatism. which would hopefully strengthen their ability to produce sound argumentative conclusions, while improving performance at distinguishing incompatible judgments and identifying self-consistent sets of normative claims. Normative reasoning frameworks could also be used to scaffold and validate the outputs of machine learned systems (Castagna et al.,, 2024), improving interpretability and correctness while still allowing the overall AI system to work with open-ended (e.g. language) inputs. Finally, one might hope to minimize the role of uninterpretable machine-learned systems altogether, using them primarily for the translation of inputs and outputs while performing most of the reasoning (normative or otherwise) via symbolic model-based algorithms (Wong et al.,, 2023; Kwon et al., 2023a, ). On this route, the main challenge will be to integrate normative reasoning with frameworks for model-based inference and planning, such as probabilistic programming (van de Meent et al.,, 2018; Cusumano-Towner et al.,, 2019).

Considerable work needs to be done before we can design AI that reasons flexibly and generally about preferences and values. Still, there exist many opportunities for research that are under-explored. By taking advantage of them, we might hope to build systems that handle the true normative complexity of the situations we are deploying them into.

Two aspects of reward functions are important for determining their role in the practice of AI alignment. The first is whether they are scalar. As explained in Section 2, this corresponds to the question of whether values are treated as fully commensurable, and whether the preferences they represent are complete. The second, often underappreciated aspect, is whether they are contextual: Is the reward function understood to be a representation of context-specific preference judgments, or of an individual’s overall preferences?

Scalar rewards are only appropriate in narrow decision contexts. Scalar rewards are generally inadequate, since (as elaborated in Section 2) they assume away the possibility of incomplete human preferences. But as long as these rewards are also understood to be contextual, then reward-based alignment can be appropriate. In relatively narrow decision contexts without sharp practical or moral dilemmas, it is not unreasonable to assume that people are willing to commensurate their values (Anderson,, 1995). In these contexts (e.g. buying groceries, travel planning, solving math homework) it is often clear to us how to weight different values against others (e.g. quality vs. cost, time vs. comfort, correctness vs. verbosity), leading to a complete preference ordering that it is representable by scalar reward. Learning a reward function is thus not inherently problematic. If this learned reward function is then optimized by a bounded AI system — the kind of local, tool-like system we discussed in Section 3 — then the downsides are also limited. A poorly learned reward function may still result in negative outcomes (Zhuang and Hadfield-Menell,, 2020), but the system will not reliably bring about unintended non-local effects.

Models of context-specific preferences will not generalize across contexts. By and large, this is the setting within which methods like RLHF are applied. Reward models are learned from human preferences, but these preferences typically represent context-specific goodness-of-a-kind judgments like “How well does this robot achieve its goal?” (Christiano et al.,, 2017) or “How well do these responses follow the provided instructions?” (Ouyang et al.,, 2022) While such judgments may implicitly aggregate a number of underlying values like “harmlessness” or “helpfulness” (Bai et al., 2022a, ), they are not judgments of goodness simpliciter, or of goodness for the user as a whole. This means that the resulting reward models are only useful for narrow alignment. They can serve as reasonable guides to in-context behavior, but are unlikely to generalize beyond that context (Lambert and Calandra,, 2023). In particular, such reward models do not represent human preferences across contexts, over an extended period of time.

Context-sensitive preference models as an intermediate solution. What would it take to align an AI system that operates across contexts? One option is the use of context-sensitive reward functions (Pitis et al.,, 2024), as described in Section 3. Though this approach runs the risk of context-manipulating incentives, it may well be adequate for sufficiently bounded systems. Similar to our ECD proposal in Section 2, context-sensitivity could be achieved by per context commensuration of multiple values, perhaps by learning separate reward or preference models for each value (Wu et al.,, 2024; Go et al.,, 2024; Xu et al.,, 2024), then aggregating their rewards with different weights depending on the downstream context. Context switches could then be triggered by users by selecting a desired “mode” (Edwards,, 2023) or specifying a system prompt (Pitis et al.,, 2024).

Still, all that the above amounts to is solving several instances of the narrow alignment problem, then stitching together the answers. If society is on a path towards more general AI systems — say, the globally-scoped AI assistants we discussed in Section 3 — then we will need more general solutions.

How should one build an AI system that is aligned not to a user in a particular context, but to assist a person over an extended period of time? Addressing this challenge requires a significantly more ambitious solution to the value alignment problem — one that not only avoids the pathologies of expected utility maximization across global scopes (cf. Section 3), but also accounts for the dynamically and socially constructed nature of our preferences.³⁰³⁰30Of course, it is always an option to avoid taking up this challenge; there are many transformative uses of AI that do not involve globally scoped personal assistance. Nonetheless, if AI researchers do aim for something like this goal, they should be clear about what it requires.

Most alignment methods do not adequately account for these aspects of preference construction. Instead, they assume that elicited preferences are static — they do not change over time — and asocial — they are independent of other agent’s preferences and societal norms. These are reasonable assumptions if AI systems are only interacting with users over relatively short timescales, and if such interactions can be decoupled from their wider social context. Unfortunately, neither of these assumptions is true in general.

Preferences change via adaptation, drift, learning, reflection, or volition. Contra the first assumption, preferences are dynamic: They change, shift, and grow over time (Franklin et al.,, 2022). This is partly the result of context, as we have discussed, and partly a feature of human psychology: per Kahneman, our stated preferences about an experience can vary with the time of elicitation (Kahneman and Riis,, 2005); per Sen and Nussbaum, our preferences adapt to the conditions of what is available to us (Sen et al.,, 1999; Nussbaum,, 2001). More generally preference change is the result of being agents who learn about the world and ourselves as we grow (Loewenstein and Angner,, 2003), and who reflect upon and reconsider what we value and desire. As we change our beliefs about what is true, what we find instrumentally valuable changes accordingly. As we discover what we experience as pleasant or unpleasant, what we consider to be intrinsically valuable may also change. We can also voluntarily change our values (Ammann,, 2023), perhaps by practicing an art form so that we may appreciate it better, or by adopting a new way of life (Chang,, 2009; Paul,, 2014).

Alignment with informed preferences as a partial solution. Can standard techniques for AI alignment be transplanted to the dynamic context? One modification is to assume that preference change is due only to people learning about their desires over time. In this model, there is still a true underlying preference structure, albeit one initially unknown to the human, and the AI system can just treat those preferences as the target of alignment (Chan et al.,, 2019). Similar modifications can be applied to the case of changing empirical beliefs: Instead of satisfying a person’s revealed preferences, the AI system aims to satisfy what their preferences would be if they were more informed (Reddy et al.,, 2018). This idea might even be extended to encompass reflection upon preferences and values (Cath,, 2016): By modeling people as bounded reasoners (Zhi-Xuan et al.,, 2020; Alanqary et al.,, 2021), and integrating such models with frameworks for normative reasoning, AI systems could infer what people would come to want, if they thought harder about what they truly value.

The challenge of genuine value change. However, alignment with informed preferences avoids the deeper normative questions raised by genuine value change: How should an AI system assist someone whose informed preferences change over time due to drift, volition, or transformation? Or what if a person’s preferences adapt in response to (potentially oppressive or addictive) environments (Sen et al.,, 1999; Nussbaum,, 2001)? Unlike preference change due to learning or reasoning, there is no sense in which the resulting preferences are more informed or “rational” than they were before. Perhaps AI systems could optimize for a person’s current preferences, but this risks shifting or manipulating their preferences in undesirable ways (Ashton and Franklin,, 2022; Carroll et al.,, 2022, 2024). Avoiding such shifts would require delineating the kinds of value change that are legitimate or illegitimate (Ammann,, 2023), but as Carroll et al., (2024) discuss, it is not obvious how to do so. Alternatively, one might hope to aggregate preferences across the time-slices that make up a person (Hedden,, 2015), but this introduces difficult questions about how to weight past, present, and future time-slices (Paul,, 2014; Pettigrew,, 2019), while ignoring the practical unity that individuates a person as a person (Korsgaard,, 1989; Schechtman,, 2014), not just a collection of consciousness moments.

Preferences are socially constructed. We shall return to these normative questions shortly. Before doing so, let us consider the assumption that preferences are asocial. In rational choice theory, preferences are typically understood to be an individual’s comparative judgments about the outcomes that would be best for them and them alone. These self-regarding preferences are often treated as the target of AI alignment (Hadfield-Menell et al.,, 2016; Russell,, 2019). But of course, many of our preferences are not asocial in this way. Instead, they are interdependent (Sobel,, 2005): formed not in isolation, but influenced by the preferences, values, and norms of our social and moral circles. Sometimes this influence is merely instrumental — one might prefer to follow a social norm just because it is convenient to do so. But sometimes the influence is constitutive — as in a parent’s concern for their child’s well-being, or a feminist’s desire to uphold a norm of equality. If we are to align an AI system with an individual, we will need some way of accounting for these influences.

Recursive preference modeling as a partial solution. As an intermediate solution to the challenge of socially constructed preferences, one might hope to align AI systems with recursive or interdependent preferences — preferences which depend on the preferences of others (Sobel,, 2005). Such preferences can be modeled with recursive utility functions, which assign weight to the posited utility functions of other agents (Kleiman-Weiner et al.,, 2017; Kim et al.,, 2018), or more general models of preference interdependence (Yang and Allenby,, 2003). Preferences or “rewards” can also depend on social and moral norms (Bicchieri,, 2005; Oldenburg and Zhi-Xuan,, 2024), reflecting how people predict and respond to the normative infrastructure of their society (Hadfield-Menell and Hadfield,, 2018).

Yet, by keeping preferences or utility functions as the target of alignment, recursive preference modeling still faces the many of the limitations we have surveyed. In particular, it still runs the risk of treating preferences as normatively basic, rather than the values and norms that generate those preferences. It also limits our ability to reason about such values and principles, and whether they are appropriately influencing an individual’s preferences. After all, many social norms and influences are oppressive or otherwise undesirable (Lukacs and Livingstone,, 1972; Althusser et al.,, 2006), shaping preferences in ways we intuitively regard as contrary to an individual’s best interests. In this sense, the problem of interdependent preferences is similar to the problem of dynamic preferences. In both cases, a range of preference orderings are at play, and without additional normative considerations, it is not clear which set of preferences an AI system should be aligned with (Carroll et al.,, 2024).

In light of the challenges introduced by contextual, dynamic, and interdependent preferences, it is difficult to see how they can serve as a coherent alignment target. This also follows from our discussion in Sections 2 and 3: If preferences are neither psychologically nor normatively basic, then it is not clear what justifies their being the target of value learning and alignment.

Alignment with role-specific normative criteria. This basic point, of course, is not new: As many have long appreciated, identifying someone’s welfare or best interests with their preferences runs into a thicket of philosophical issues (Sen et al.,, 1999; Nussbaum,, 2001). Recognizing these issues, Gabriel, (2020) argues for an explicitly moral conception of alignment: “the agent does what it morally ought to do, as defined by the individual or society”.³¹³¹31Gabriel, (2020) uses “values” to describe this alignment target, though in a slightly narrower sense than ours. Whereas we have primarily used “values” to refer to evaluative concepts and judgments in general, Gabriel’s use implicitly picks out the values that are normatively relevant to AI system behavior. Others have proposed similar approaches, though they replace “morally ought” with what an agent or humanity as a whole would reflectively endorse, as in ideal observer theories (Firth,, 1952; Brandt,, 1955) or coherent extrapolated volition (Yudkowsky,, 2004). However, it is far from clear how to operationalize these abstract principles. To make progress, we suggest a conception of single-principal alignment that is significantly more constrained: When an AI system only serves an individual in performing a particular task or role, it should be aligned with the normative ideals or criteria that are appropriate for that role. For narrow systems, this requires task-specific determination of appropriate normative criteria. For general-purpose AI assistants, this implies alignment with the normative ideal of an assistant, rather than alignment to an individual’s preferences, or to human normativity writ large.

Existing methods effectively align AI with role-specific norms. Before discussing the case of general-purpose assistants, it is worth noting that many existing alignment methods effectively function to align AI systems with task and role-specific norms, even though they are described as methods for alignment with human preferences.³²³²32This preferentist focus is explicit in e.g. Ouyang et al., (2022), who introduce an application of RLHF to LLMs that, in their words, “aligns the behavior of GPT-3 to the stated preferences of a specific group of people.” As discussed earlier, the pairwise judgments provided by human annotators in RLHF are typically not their preferences as end users, but instead context-specific goodness-of-a-kind judgments. These judgments are provided in response to questions about whether an AI system’s output complies with specific normative criteria — for example, helpfulness, harmlessness, and truthfulness (Ouyang et al.,, 2022; Bai et al., 2022a, ). As such, insofar as these judgments can be called preferences, they are derivative of normative standards like harmlessness, not the alignment target themselves. Preferences merely serve as data so that machines can learn some approximation of these standards. The typical language used to describe reward-learning methods like RLHF is thus misconceived: As used, they are not methods for alignment with any one human’s preferences, or for recovering the “true reward function” in some person’s head, ³³³³33Supposing the concept of a “human reward function” is even coherent. See Butlin, (2021) for a discussion. but for aligning AI systems with contextually-appropriate normative criteria.

Normative criteria for general purpose AI assistants. What then are the normative criteria for general-purpose AI assistants — those globally scoped AI systems for which questions of preference change and incompleteness seem the most pressing? While we cannot give a definitive answer — indeed, as we shall discuss, we think this is something that society will have to collectively decide — we suggest that progress can be made by reflecting on the normative ideal of a good assistant.

How does this ideal address the issues with preference alignment that we have raised? Here are a number of suggestions: First, a good assistant does not presume certainty about a person’s preferences and values (Hadfield-Menell et al.,, 2016). This means maintaining an awareness of their own ignorance, while avoiding unwarranted extrapolation of preferences from one context to another, including Knightian uncertainty about how preferences extrapolate (Dalrymple et al.,, 2024). Second, a good assistant is aware that some choices are hard, and some options may seem incomparable (Chang,, 1997). When helping someone with such a choice, the assistant does not pretend to know which option is better, or try to optimize that person’s life; instead, the assistant respects their autonomy, and empowers them to make the most informed choice possible (Du et al.,, 2020), while ultimately remaining agnostic as to which choice is “best”. Third, a good assistant understands and respects the values of the person they are assisting. This means recognizing that a person’s preferences often derive from their values, which can take priority over their immediate requests and preferences (London and Heidari,, 2024). The assistant also enables those values to grow and change through normatively acceptable forms of exploration, reflection, volition, or even drift, while avoiding manipulating them or restricting them (Ammann,, 2023). Finally, a good assistant, being situated in wider society, respects the preferences and values of others (Kirk et al.,, 2024). When assisting someone who wishes harm out of anger, the assistant might dissuade them from acting against their better nature. When asked to directly harm others, the assistant might refuse.

Pathways to aligning general purpose assistants. In a past era of AI development, these principles might have seemed too vague to formalize or implement. Yet, as our discussion of RLHF suggests, it now seems like we have at least one path towards aligning globally-scoped AI assistants: Train them to comply with human judgments and standards for ideal assistive behavior. Methods such as harmless and helpful RLHF (Bai et al., 2022a, ), (collective) constitutional AI (Bai et al., 2022b, ; Huang et al.,, 2024), and moral graph elicitation (Klingefjord et al.,, 2024) are already taking steps in this direction, each of them making more explicit that the targets of alignment are not preferences, but normative principles for assistance. Such systems still have to learn the preference of each user they assist, but this is separate from learning how to provide assistance in light of those preferences.

Within this broad approach, we can embed many of the proposals we have made in earlier sections. Rich but structured models of human decision-making can serve as the AI assistant’s “theory-of-mind”, producing well-calibrated estimates of user goals and preferences while avoiding the deficiencies of unstructured approaches (Zhi-Xuan et al., 2024b, ; Kim et al.,, 2023). Mechanisms for preference incompleteness could be engineered or trained into the AI assistant if this turns out to remove incentives for shutdown avoidance and context manipulation (Thornley,, 2024). Theories of normative reasoning could be integrated into AI systems, allowing them to reason about human-provided judgments and principles, while aiding us in deliberating about what counts as good assistance. Each of these proposals may turn out to be strictly unnecessary for the task. Even so, they can provide us helpful guidance as we refine and implement our normative ideals of assistance.

Different types of preferences are subject to aggregation. Discussion of preference aggregation and its uses is often afflicted by confusion about the nature of preferences. Are these all-things-considered preferences, or goodness-of-a-kind judgments? Are these preferences over outcomes (Harsanyi,, 1953), or preferences over ethical views (Baum,, 2020)? Are these self-regarding preferences, social preferences, or some combination of the two? For clarity, we shall use the term welfare preferences (Rubinstein and Salant,, 2012) to refer to those preferences that Harsanyi’s theorem most intuitively applies to: These are self-regarding preferences over outcomes that affect one’s individual welfare, which exclude consideration of others’ welfare. We distinguish this concept from all-things-considered preferences, which are preferences about overall goodness (including social or moral considerations), and from elicited preferences, which refers to any kind of preference elicited while applying some alignment technique.

Aggregation of elicited preferences need not track aggregate welfare or goodness. The first thing to note is that elicited preferences, welfare preferences, and all-things-considered preferences may all come apart. This crucially affects why and how we aggregate preferences, and whether some utilitarian aggregation procedure should be used. Consider a hypothetical example in the context of RLHF: Users are asked whether they would personally enjoy an LLM that can generate copyrighted short stories, and most of them say yes. If what we care about is aggregate (immediate) welfare, then uniform aggregation of the elicited preferences seems to achieve that goal. But if we what we care about aggregating are all-things-considered value judgments — including legal and moral considerations — then uniform aggregation no longer seems so appropriate.

Similar issues arise when trying to aggregate toxicity or harmfulness judgments across multiple humans (Bai et al., 2022a, ; Davani et al.,, 2022). In these cases, the elicited preferences are goodness-of-a-kind judgments, and their connection to aggregate welfare (or all-things-considered goodness) is many steps removed. As such, uniform or majoritarian aggregation can easily fail to achieve social goals. If most human annotators are insensitive to certain forms of identity discrimination (e.g. sexually demeaning images, trans-exclusionary rhetoric, or anti-semitic tropes), then AI systems trained on such data will almost certainly cause harm (Richardson et al.,, 2019; Okidegbe,, 2021). Uniform preference aggregation may thus constitute a form of epistemic injustice (Fricker,, 2007; Symons and Alvarado,, 2022; Hull,, 2023), which in turn leads to downstream injustice and harm.

Non-utilitarian aggregation may be beneficial on normative or epistemic grounds. What aggregation procedures might we use instead? And what justifies their use? In the case of potential copyright violations, we might want to grant veto power to copyright holders, allowing them to reasonably reject the welfare-oriented majority preference for copying their work. This veto right could be justified as an instantiation of Scanlon’s contractualism (Scanlon,, 2000), on the principle that mutual respect among persons necessitates taking claims of intellectual ownership seriously. Alternatively, it could simply be understood as a policy that everyone would reflectively prefer, once they properly understood the costs and benefits of a copyright veto.

As for harmfulness judgments, it may often be preferable to apply prioritarian (Lumer et al.,, 2005; Holtug,, 2017) or egalitarian (Rawls,, 1971) approaches to aggregation. For example, one might select annotators who are most directly impacted by potential harms (Gordon et al.,, 2022), thereby prioritizing certain segments of the population. In cases of significant disagreement, one might even place all weight on the individual with the strongest dispreference (Leben,, 2017; Bakker et al.,, 2022; Weidinger et al.,, 2023). Again, there are many possible justifications for such procedures. Prioritarian selection could be justified on normative grounds, or because of its epistemic benefits — after all, those most impacted by harms also tend to be more informed about their effects (Dror,, 2023).

Distinguishing aggregation procedures from standards of rightness. Whatever procedure one favors, it is important not to confuse the aggregation rules used in AI systems with our ultimate social objectives. In practice, these aggregation rules are merely parts of the overall decision procedure implemented by (training) an AI system, and as many philosophers have pointed out, such procedures should be distinguished from standards of rightness (Railton,, 1993; Frazier,, 1994; Stark,, 1997). Rather than directly instantiating a particular standard (or its mathematical formalization) into a preference aggregation procedure, we should consider which aggregation procedures best satisfy the standard(s) we care about, taking into account practical and informational constraints. In doing so, we should recognize that elicited preferences are typically not the objects of our concern, but simply information as to what we truly care about.

Suppose we recognize that any particular set of elicited preferences is merely a guide or estimate to what we care about. Even so, one could still imagine taking humanity’s aggregate preferences as the target of AI alignment. For example, suppose that humanity eventually builds a single powerful AI system — a singleton — that actively infers the preferences of all humans, uses those preferences to estimate humanity’s social welfare function, then optimizes its best estimates of that function. In doing so, we might create the ideal utilitarian central planner, achieving what welfare economists and utopian socialists could only dream of (Ng,, 1997; Bastani,, 2019).

Theoretical difficulties for preference aggregation. Unfortunately, taking aggregate preferences as an alignment target immediately runs into theoretical difficulties. While these issues have been studied at length by social choice theorists,³⁵³⁵35See Baum, (2020); Korinek and Balwit, (2022); Mishra, (2023) and Conitzer et al., (2024) for discussions of the challenge of applying social choice to AI alignment. one that is especially challenging for standard utilitarian aggregation is incomparability. As we noted earlier, justifications for preference aggregation typically assume that each individual’s preferences can be represented as a utility function, and furthermore that utility can be compared across persons (Harsanyi,, 1953, 1975). But as we have elaborated Section 2, these assumptions are very much in doubt. Even within a single individual, preferences may be incomplete due to incomparable choices, or not clearly comparable across time (Carroll et al.,, 2024). Having to compare the goodness of choices across individuals only makes the difficulty more severe (Korinek and Balwit,, 2022). This is not to say that the preferability of some outcome can never be compared across people,³⁶³⁶36For example, if the choice of person A not wearing a mask would lead to less inconvenience for person A but severe illness for person B, we should intuitively give a stronger weight to person B’s preference against severe illness over person A’s preference against inconvenience. but that any such comparison stands in need of further normative justification (Sen, 1970a, ; Clayton and Williams,, 1999) — justification that, as we argued in Section 3, utility theory alone cannot provide.

The computational intractability of aggregate preference optimization. Let us suppose, however, that these theoretical challenges can be addressed.³⁷³⁷37Perhaps by using frameworks that allow for partial comparability of welfare across individuals (Sen, 1970b, ), or by aligning AI with partial social preferences (Korinek and Balwit,, 2022). Even so, aggregate preference optimization still faces serious practical challenges. For one, such optimization is computationally intractable: As Austrian economists have long argued, central planning runs into the economic calculation problem (von Mises,, 1990), a problem made worse by the sheer complexity of inferring human preferences under limited information, coordinating global production to maximize aggregate preferences, and planning for the future under uncertainty (Hayek,, 1945; Murphy,, 2006; Cwik and Engelhardt,, 2024).³⁸³⁸38These difficulties can be formalized with the aid of theoretical computer science, which shows that optimal planning under uncertainty is sometimes undecidable, and even when decidable, remains anywhere from PSPACE to EXPTIME-complete (Papadimitriou and Tsitsiklis,, 1987; Chatterjee et al.,, 2016). In contrast, decentralized decision-making (in the form of e.g. competitive markets) can sometimes be exponentially more efficient in computational cost than central planning (Rust,, 1996), while achieving optimal informational efficiency (Mount and Reiter,, 1974; Jordan,, 1982). As such, even if not a practical impossibility, optimizing humanity’s aggregate preferences with a single AI system is likely to be considerably less efficient than more pluralistic alternatives (Siddarth et al.,, 2022).

The politically infeasibility of impartially benevolent AI. Perhaps even more importantly, the project of building AI that optimizes humanity’s aggregate preferences is politically infeasible: Even if impartially benevolent AI planners were possible to develop, building such systems would be incompatible with the incentives of every AI developer with a realistic chance of doing so. This is the case even for AI developers with expressedly pro-social missions, which are still subject to market incentives as a result of the need to raise capital (Toner and McCauley,, 2024), and are still governed by the laws and regulations of the countries they are based in. Allowing the creation of such AI systems would also risk the centralization of immense power: However virtuous the goal of impartial preference optimization might seem, the history of central planning should tell us that optimal social outcomes are far from likely to be achieved (Scott,, 1998; Verdery,, 2005). Instead, we are more likely to see a tyranny of creator values, with potentially disastrous consequences for everyone with a contrary way of life.

Pluralistic alignment as a politically feasible alternative. In light of these challenges, how should we reconceive the goals of multi-principal AI alignment? One constraint in doing so is incentive compatibility: Whatever our vision of AI alignment is, it should account for divergent interests and contested values, credibly enabling collective safety and stability by ensuring incentives for cooperation and minimizing the chances of conflict (Critch and Krueger,, 2020; Dafoe et al.,, 2020). A related constraint is political feasibility: Alternative targets for alignment should be achievable given the political economy of actually existing AI — an economy that consists of a wide variety of AI services developed and deployed by a large number of self-interested actors (Drexler,, 2019). Although these are negative constraints, they pair well with a more positive, pluralistic vision of what alignment could enable: A world where increasingly advanced AI systems serve a diversity of individual, communal, and universal ends, without catastrophically endangering anyone’s interests (Zhi-Xuan,, 2022; Gabriel,, 2020; Siddarth and Huang,, 2023).

Enabling pluralism through political constraints. What would it require to enact this pluralistic vision? As a starting point, consider our principles for AI assistance from Section 4. While an AI assistant primarily serves a single person, and might be personalized to do so in many ways (Sorensen et al.,, 2024), our presumptive norms for ideal assistance do not permit disregard for others. Rather, they endorse a circumscribed promotion of the person’s interests and values, such that the assistant avoids harming and endangering other individuals.³⁹³⁹39This might be viewed as an instantiation of the Harm Principle (Mill,, 1859) for AI assistants. These norms function as political constraints, allowing assistants to provide value for individual users without imposing unreasonable externalities upon others.⁴⁰⁴⁰40See also Kirk et al., (2024) on the bounds of personalization in LLM assistants, and Gabriel and Keeling, (2024) for an explicitly political conception of AI alignment. In doing so, they reduce the chance of conflict and non-cooperation.

Alignment with politically negotiated normative standards. Our suggestion then, is that this approach can be generalized to broadly contractualist account of AI alignment (Zhi-Xuan,, 2022):⁴¹⁴¹41We use “contractualist” here in a broad sense, which includes both contractarianism (Cudd and Eftekhari,, 2021) and Rawlsian (Rawls,, 1971) or Scanlonian contractualism (Scanlon,, 2000). Rather than learning humanity’s preferences in order to maximally satisfy them, AI systems should be aligned with normative standards and criteria that we collectively forge and negotiate — standards exemplified by social, legal, and moral norms. These norms may be constructed as we design each system, or can be decided in advance for entire classes of AI systems. Returning to our earlier discussion of role-specific alignment, what is important is that these norms are tailored to the scope and uses of each system: Just as AI assistants should avoid harmful language, self-driving cars should follow the rules of the road. By negotiating norms and constraints for each of AI’s social functions, we can enable a plurality of uses for AI while limiting the costs and harms to all stakeholders involved.

The practical benefits of contractualist alignment. What benefits does a contractualist approach to alignment offer? In our view, its primary benefits are practical ones: Unlike aggregate preference optimization, contractualist alignment does not require unrealistic amounts of benevolence from any one actor. Instead, it aims for a regime where largely self-interested actors stand to mutually benefit from the development and deployment of AI. Well-designed norms and institutions enable this, stabilizing cooperation by making it costly for relevant parties to defect or withdraw from cooperation (Kalai and Smorodinsky,, 1975; Gintis,, 2010). Aligning AI systems to comply with cooperative norms (and perhaps even to enforce them) thus reduces the chance of AI-caused or mediated conflict, or the risk of (catastrophically) endangering anyone’s interests. Norms also limit the computational and informational cost of ensuring aligned behavior: Rather than inferring a large number of preferences, norm-aligned agents just have to (learn to) comply with a limited set of constraints (Oldenburg and Zhi-Xuan,, 2024). Finally, by centering norms and principles as the targets of AI alignment, political deliberation becomes more feasible and widely accessible (Huang et al.,, 2024): Stakeholders need not negotiate over every last detail over how an AI system is built or trained, but can instead agree upon high-level requirements and standards for how the system should behave.⁴²⁴²42This does not preclude lower-level forms feedback such as participatory data labeling (Gordon et al.,, 2022) or end-user audits (Lam et al.,, 2022), which can complement the aim of mutually-acceptable AI design.

The normative grounds of contractualist alignment. Besides its practical benefits, contractualist alignment can also be grounded in normative foundations that are more compatible with a pluralistic world. While it might be possible to justify broadly contractualist principle-setting on rule consequentialist grounds (Parfit,, 2011), we contend that the normative appeal of contractualist alignment is precisely that it avoids a universal account of what consequences are better or worse.⁴³⁴³43Similar arguments are made by Gabriel, (2020) and Gabriel and Keeling, (2024). Given the difficulties with comparability that we have examined, it is unlikely that people will ever agree upon a single scale of value for ranking all consequences. Instead, contractualist alignment aims to align AI systems with goals, standards, and principles that are mutually agreed upon by people despite our disparate preferences and values, deriving its normative force from the fair and impartial agreement of relevantly-situated rational actors.

Conditions for fair and impartial agreement. What makes an agreement impartial or fair? As in contractarian moral and political theories (Gauthier,, 1986; Binmore,, 1994), it may be enough that all stakeholders benefit relative to an originally fair bargaining position, subject to additional symmetry constraints. Or as Rawls, (1993) and Scanlon, (2000) respectively argue, a thicker conception of public reason and the mutual recognition of each other as reasonable persons may be necessary to decide which agreements are fair. While examining these questions would take us beyond the scope of this paper, we believe our critique of expected utility theory lends itself to thicker conceptions of fair and reasonable agreement. On such conceptions, AI systems should not just be aligned with goals and standards that achieve mutual benefit.⁴⁴⁴⁴44After all, mutual benefit is not always achievable. In such cases, it is still possible to reach agreements that are viewed as fair, as in an agreement to compensate someone for harm. Instead, AI goals and standards should be justified to each stakeholder, on grounds that none can reasonably reject. Insofar as these AI systems are used to exercise power over others, they should also act in accordance with standards that are not just fair, but legitimate (Lazar,, 2024; Stone and Mittelstadt,, 2024).

Alignment in the absence of agreement. A natural worry for contractualist alignment is the possibility that agreement between different stakeholders may not be obtained (let alone agreement that is impartial and fair). Yet, this worry is not as acute as it may initially seem. First, rather than aligning AI systems with norms that have actually been agreed upon, we could align them with norms that would hypothetically be agreed upon, in the spirit of virtual bargaining (Misyak et al.,, 2014; Chater,, 2023). This would generally be necessary to handle incompletely specified agreements and contracts (Hadfield-Menell and Hadfield,, 2018), while sharply lowering the cost and frequency of actual negotiations. Second, there are many cases where the operation of an AI system imposes minimal externalities upon others, and hence the cost of disagreement between AI stakeholders is merely that the gains of cooperation cannot be realized. In such cases, it is no great loss if each party operates their own AI system aligned with their individual goals, rather than having a shared AI system aligned with collective goals and norms. It is only when AI systems do impose substantial negative externalities that disagreement about their operation is more dangerous. These situations could well lead to mutually destructive conflict, as in prisoner’s dilemma scenarios, or exploitative outcomes, where some AI operators benefit significantly at the expense of others. Even so, humans still have the political agency to shape which agreements are feasible and fair, and there is reason to hope that parties will negotiate to avoid at least the worst AI outcomes (e.g. in the form of minimal safety standards). Finally, achieving agreement over norms and principles is likely to be far easier than agreeing on a metric for globally ranking all consequences or comparing all people’s preferences. As such, unless one is willing to allow a small set of actors to decide how all of humanity’s preferences should be weighted and compared, utilitarian preference aggregation faces an even sharper risk of disagreement and conflict than a contractualist approach.

Technical avenues toward contractualist alignment. If we accept this contractualist understanding of multi-principal alignment, then much work remains to be done. On the technical front, there need to be advances in the theory and implementation of cooperative or contractualist decision-making. While recent alignment techniques show how language-based AI assistants can be aligned with collectively elicited norms and values, and how divergences in norms, opinions, and values can be reconciled through agreement (Huang et al.,, 2024), iterative critique (Bakker et al.,, 2022), or moral reflection (Klingefjord et al.,, 2024), these methods are specialized to a particular type of AI system, and have yet to be situated in a more general theoretical framework. To develop such a framework, we suspect that it will be necessary to unite ideas from game theory (Dafoe et al.,, 2020), bargaining theory (Chater,, 2023), and social choice (Conitzer et al.,, 2024) with formal approaches to argumentation (Amgoud and Cayrol,, 1998) and negotiation (Rahwan et al.,, 2003), along with insights from the science of human normativity (Binmore,, 1994; Hadfield-Menell and Hadfield,, 2018; Levine et al.,, 2023). In particular, by developing computational theories of how humans rapidly learn extant norms and conventions (Tan and Ong,, 2019; Hadfield-Menell et al.,, 2019; Hawkins et al.,, 2019), recognize institutional structure (Jara-Ettinger and Dunham,, 2024; Baker et al.,, 2024), and engage in contractualist reasoning about social and moral norms (Levine et al.,, 2023, 2024), we can inform the design of AI systems with social and normative competence: AI that is not just aligned with stakeholder values in a once-off process, but which flexibly adapts to our norms and institutions as they evolve (Oldenburg and Zhi-Xuan,, 2024), reasons about their applicability in novel situations (Kwon et al., 2023b, ), and perhaps even aids us in negotiating new contracts and norms (Christoffersen et al.,, 2023; Jarrett et al.,, 2023; Tessler et al.,, 2024).

Social and political avenues toward contractualist alignment. Of course, if we take fair and impartially negotiated standards as the target of AI alignment, then technical advances will not be enough; we also need to foster the development of social, economic, and political orders that provide the conditions for free and fair agreement. This might involve the creation of new economic and political mechanisms that elicit and consolidate the interests of AI stakeholders (Siddarth and Huang,, 2023), the establishment of democratic processes and bodies that can exercise legitimate authority over AI systems (Ovadya,, 2023), or the expansion of participatory approaches to AI development and design (Birhane et al.,, 2022; Suresh et al.,, 2024). Without these social and political investments, we will lack the capacity to surface our reasons and values to AI systems that act on our behalf, and the accountability to ensure that each of our interests is fairly represented. After all, if we are going to align AI systems with normative standards we would collectively endorse, then we had better make sure that a “we” exists to endorse them.