Policy Aggregation

Early discussion of AI value alignment had often focused on learning desirable behavior from an individual teacher, for example, through inverse reinforcement learning [27, 1]. But, in recent years, the conversation has shifted towards aligning AI models with large groups of people or even entire societies. This shift is exemplified at a policy level by OpenAI’s “democratic inputs to AI” program [41] and Meta’s citizens’ assembly on AI governance [8], and at a technical level by the ubiquity of reinforcement learning from human feedback [30] as a method for fine-tuning large language models.

We formalize the challenge of value alignment with multiple individuals as a problem that we view as fundamental — policy aggregation. Our starting point is the common assumption that the environment can be represented as a Markov decision process (MDP). While the states, actions and transition functions are shared by all agents, their reward functions — which incorporate values, priorities or subjective beliefs — may be different. In particular, each agent has its own optimal policy in the underlying MDP. Our question is this: How should we aggregate the individual policies into a desirable collective policy?

A naïve answer is to define a new reward function that is the sum of the agents’ reward functions (for each state-action pair separately) and compute an optimal policy for this aggregate reward function; such a policy would guarantee maximum utilitarian social welfare. This approach has a major shortcoming, however, in that it is sensitive to affine transformations of rewards, so, for example, if we doubled one of the reward functions, the aggregate optimal policy may change. This is an issue because each agent’s individual optimal policy is invariant to (positive) affine transformations of rewards, so while it is possible to recover a reward function that induces an agent’s optimal policy by observing their actions over time,¹¹1And we assume this is done accurately, in order to focus on the essence of the policy aggregation problem. it is impossible to distinguish between reward functions that are affine transformations of each other. More broadly, economists and moral philosophers have long been skeptical about interpersonal comparisons of utility [19] due to the lack of universal scale — an issue that is especially pertinent in our context. Therefore, aggregation methods that are invariant to affine transformations are strongly preferred.

Our approach. To develop such aggregation methods, we look to social choice theory, which typically deals with the aggregation of ordinal preferences. To take a canonical example, suppose agents report rankings over $m$ alternatives. Under the Borda count rule, each voter gives $m-k$ points to the alternative they rank in the $k$’th position, and the alternative with most points overall is selected.

The voting approach can be directly applied to our setting. For each agent, it is (in theory) possible to compute the value of every possible (deterministic) policy, and rank them all by value. Then, any standard voting rule, such as Borda count, can be used to aggregate the rankings over policies and single out a desirable policy. The caveat, of course, is that this method is patently impractical, because the number of policies is exponential in the number of states of the MDP.

The main insight underlying our approach is that ordinal preferences over policies have a much more practical volumetric interpretation in the state-action occupancy polytope $\mathcal{O}$. Roughly speaking, a point in the state-action occupancy polytope represents a (stochastic) policy through the frequency it is expected to visit different state-action pairs. If a policy is preferred by an agent to a subset of policies $\mathcal{O^{\prime}}$, its “rank” is the volume of $\mathcal{O^{\prime}}$ as a fraction of the volume of $\mathcal{O}$. The “score” of a policy under Borda count, for example, can be interpreted as the sum of these “ranks” over all agents.

Our results. We investigate two classes of rules from social choice theory, those that guarantee a notion of fairness and voting rules. By mapping ordinal preferences to the state-action occupancy polytope, we adapt the different rules to the policy aggregation problem.

The former class is examined in Section 5. As a warm-up we start from the notion of proportional veto core; it follows from recent work by Chaudhury et al. [7] that a volumetric interpretation of this notion is nonempty and can be computed efficiently. We then turn to quantile fairness, which was recently introduced by Babichenko et al. [4]; we prove that the volumetric interpretation of this notion yields guarantees that are far better than those known for the original, discrete setting, and we design a computationally efficient algorithm to optimize those guarantees.

The latter class is examined in Section 6; we focus on volumetric interpretations of $\alpha$-approval (including the ubiquitous plurality rule, which is the special case of $\alpha=1$) and the aforementioned Borda count. In contrast to the rules studied in Section 5, existence is a nonissue for these voting rules, but computation is a challenge, and indeed we establish several computational hardness results. To overcome this obstacle, we implement voting rules for policy aggregation through mixed integer linear programming, which leads to practical solutions.

Finally, our experiments in Section 7 evaluate the policies returned by different rules based on their fairness; the results identify quantile fairness as especially appealing. The experiments also illustrate the advantage of our approach over rules that optimize measures of social welfare (which are sensitive to affine transformations of the rewards).

Noothigattu et al. [28] consider a setting related to ours, in that different agents have different reward functions and different policies that must be aggregated. However, they assume that the agents’ reward functions are noisy perturbations of a ground-truth reward function, and the goal is to learn an optimal policy according to the ground-truth rewards. In social choice terms, our work is akin to the typical setting where subjective preferences must be aggregated, whereas the work of Noothigattu et al. [28] is conceptually similar to the setting where votes are seen as noisy estimates of a ground-truth ranking [39, 9, 6].

Chaudhury et al. [7] study a problem completely different from ours: fairness in federated learning. However, their technical approach served as an inspiration for ours. Specifically, they consider the proportional veto core and transfer it to the federated learning setting using volumetric arguments, by considering volumes of subsets in the space of models. Their proof that the proportional veto core is nonempty carries over to our setting, as we explain in Section 5.

There is a body of work on multi-objective reinforcement learning (MORL) and planning that uses a scalarization function to reduce the problem to a single-objective one [32, 21]. Other solutions to MORL focus on developing algorithms to identify a set of policies approximating the problem’s Pareto frontier [38]. A line of work more closely related to ours focuses on fairness in sequential decision making, often taking the scalarization approach to aggregate agents’ preferences by maximizing a (cardinal) social welfare function, which maps the vector of agent utilities to a single value. Ogryczak et al. [29] and Siddique et al. [34] investigate generalized Gini social welfare, and Mandal and Gan [25], Fan et al. [13] and Ju et al. [23] focus on Nash and egalitarian social welfare. Alamdari et al. [3] study this problem in a non-Markovian setting, where fairness depends on the history of actions over time, and introduce concepts to assess different fairness criteria at varying time intervals. A shortcoming of these solutions is that they are not invariant to affine transformations of rewards — a property that is crucial in our setting, as discussed earlier.

Our work is closely related to the pluralistic alignment literature, aiming to develop AI systems that reflect the values and preferences of diverse individuals [35, 10]. Alamdari et al. [2] propose a framework in reinforcement learning in which an agent learns to act in a way that is considerate of the values and perspectives of humans within a particular environment. Concurrent work explores reinforcement learning from human feedback (RLHF) from a social choice perspective, where the reward model is based on pairwise human preferences, often constructed using the Bradley-Terry model [5]. Zhong et al. [42] consider the maximum Nash and egalitarian welfare solutions, and Swamy et al. [36] propose a method based on maximal lotteries due to Fishburn [14].

For $t\in\mathbb{N}$, let $[t]=\{1,2,\ldots,t\}$. For a closed set $S$, let $\Delta(S)$ denote the probability simplex over the set $S$. We denote the dot product of two vectors as $\langle x,y\rangle=\sum_{i=1}^{d}x_{i}\cdot y_{i}$ for $x,y\in\mathbb{R}^{d}$. A halfspace in $\mathbb{R}^{d}$ determined by $w\in\mathbb{R}^{d}$ and $b\in\mathbb{R}$ is the set of points satisfying $\{x\in\mathbb{R}^{d}\mid\langle x,w\rangle\leqslant b\}$. A polytope $\mathcal{O}\subseteq\mathbb{R}^{d}$ is the intersection of a finite number of halfspaces, i.e., a convex subset of the $d$-dimensional space $\mathbb{R}^{d}$ determined by a set of linear constraints $\{x\mid Ax\leqslant b\}$ where $A\in\mathbb{R}^{k\cdot d}$ is a matrix of coefficients of $k$ linear inequalities and $b\in\mathbb{R}^{k}$.

In a MOMDP $M$, each agent $i$ incorporates their values and preferences into their respective reward function $R_{i}$. Agent $i$ prefers $\pi$ over $\pi^{\prime}$ if and only if $\pi$ achieves higher expected return, $J_{i}(\pi)>J_{i}(\pi^{\prime})$, and is indifferent between two policies $\pi$ and $\pi^{\prime}$ if and only if $J_{i}(\pi)=J_{i}(\pi^{\prime})$. As discussed before, given a state-action occupancy measure $d_{\pi}$ in the state-action occupancy polytope $\mathcal{O}$, we can recover the corresponding policy $\pi$. Therefore, we can interpret $\mathcal{O}$ as the domain of all possible alternatives over which the $n$ agents have heterogeneous weak preferences (with ties). Agent $i$ prefers $d_{\pi}$ to $d_{\pi^{\prime}}$ in $\mathcal{O}$ if and only if they prefer $\pi$ to $\pi^{\prime}$. We study the policy aggregation problem through this lens; specifically, we design or adapt voting mechanisms where the (continuous) space of alternatives is $\mathcal{O}$ and agents have weak preferences over them determined by their reward functions $R_{i}$.

Affine transformation and reward normalization. A particular benefit of this interpretation, as mentioned before, is that all positive affine transformations of the reward functions, i.e., $aR_{i}+b$ for all $a\in\mathbb{R}_{\geqslant 0}$ and $b\in\mathbb{R}$, yield the same weak ordering over the polytope. Hence, we can assume without loss of generality that $J_{i}(\pi)\in[0,1]$. Further, we can ignore agents that are indifferent between all policies, i.e., $\min_{\pi}J_{i}(\pi)=\max_{\pi}J_{i}(\pi)$, and normalize reward functions $R_{i}\leftarrow\frac{R_{i}-\min_{\pi}{J_{i}(\pi)}}{\max_{\pi}{J_{i}(\pi)}-\min_{\pi}J_{i}(\pi)}$ such that $\min_{\pi}J_{i}(\pi)=0$ and $\max_{\pi}J_{i}(\pi)=1$. The relative ordering of the policies does not change since for all points $d_{\pi}\in\mathcal{O}$ we have $\sum_{s,a}d_{\pi}(s,a)=1$.

Volumetric definitions. A major difference between voting over a continuous space of alternatives and the classical voting setting is that the domain of alternatives is infinite and not all voting mechanisms can be directly applied to the policy aggregation problem. In particular, various voting rules require reasoning about the rank of an alternative or the size of some subset of alternatives. For instance, the Borda count of an alternative $c$ over a finite set of alternatives is defined as the number (or fraction) of candidates ranked below $c$. In the continuous setting, for almost all of the mechanisms that we discuss later, we use the measure or volume of a subset of alternatives to refer to their size. For a measurable subset $\mathcal{O}^{\prime}\subseteq\mathcal{O}$, let ${\rm{vol}}(\mathcal{O}^{\prime})$ denote its measure. The ratio ${\rm{vol}}(\mathcal{O}^{\prime})/{\rm{vol}}(\mathcal{O})$ is the fraction of alternatives that lie in $\mathcal{O}^{\prime}$. A probabilistic interpretation is that for a uniform distribution over the polytope $\mathcal{O}$, ${\rm{vol}}(\mathcal{O}^{\prime})/{\rm{vol}}(\mathcal{O})$ denotes the probability that a policy uniformly sampled from $\mathcal{O}$ lies in $\mathcal{O}^{\prime}$. We can also define the expected return distribution of an agent over $\mathcal{O}$ as a random variable that maps a policy to its expected return, i.e., one that maps $d_{\pi}\in\mathcal{O}$ to $\langle d_{\pi},R_{i}\rangle$. The pdf and cdf of this r.v. is defined below.

A useful observation about $f_{i}$, the pdf, is that it is unimodal, i.e., increasing up to its mode $\mathrm{mode}(f_{i})\in\operatorname{\arg\max}_{v\in\mathbb{R}}f_{i}(v)$ and decreasing afterwards, which follows from the Brunn–Minkowski inequality [17]. Since $f_{i}$ (pdf) is the derivative of $F_{i}$ (cdf), the unimodality of $f_{i}$ implies that $F_{i}$ is a convex function in $(-\infty,\mathrm{mode}(f_{i})]$ and concave in $[\mathrm{mode}(f_{i}),\infty)$.

In our algorithms, we use a subroutine that measures the volume of a polytope, which we denote by $\mathrm{vol\text{-}comp}(\{Ax\leqslant b\})$. Dyer et al. [12] designed a fully polynomial time randomized approximation scheme (FPRAS) for computing the volume of polytopes. We report the running time of algorithms in terms of the number of calls to this oracle.

In this section, we utilize the volumetric interpretation of the state-action occupancy polytope to extend fairness notions from social choice to policy aggregation, and we develop algorithms to compute stochastic policies provably satisfying these notions.

In this section, we adapt existing voting rules from the discrete setting to policy aggregation and discuss their computational complexity.

Plurality. The plurality winner is the policy that achieves the maximum number of plurality votes or “approvals,” where agent $i$ approves a policy $\pi$ if it achieves their maximum expected return $J_{i}(\pi)=\max_{\pi^{\prime}}J_{i}(\pi^{\prime})=1$. Hence the plurality winner is a policy in $\operatorname{\arg\max}_{\pi}\sum_{i\in[n]}\operatorname{\mathbb{I}}[J_{i}(\pi)=1]$. This formulation does not require the volumetric interpretation. However, in contrast to the discrete setting where one can easily count the approvals for all candidates, we show that solving this problem in the context of policy aggregation is not only $\mathbf{NP}$-hard, but hard to approximate up to factor of a $1/n^{1/2-\epsilon}$. We establish the hardness of approximation by a reduction from the maximum independent set problem [20]; we defer the proof to Appendix A.

Nevertheless, we can compute plurality in practice, as we discuss below.

$\alpha$-approval. We extend the $k$-approval rule using the volumetric interpretation of the occupancy polytope, similarly to the $q$-quantile fairness definition. For some $\alpha\in[0,1]$, agents approve a policy $\pi$ if its return is among their top $\alpha$ fraction of $\mathcal{O}$, i.e., $F_{i}(J_{i}(\pi))\geqslant\alpha$. The $\alpha$-approval winner is a policy that has the highest number of $\alpha$-approvals, so it is in $\operatorname{\arg\max}_{\pi}\sum_{i\in[n]}\operatorname{\mathbb{I}}[F_{i}(J_{i}(\pi))\geqslant\alpha]$. Note that plurality is equivalent to $1$-approval. It is worth mentioning that there can be infinitely many policies that have the maximum approval score and, to avoid a suboptimal decision, one can return a Pareto optimal solution among the set of $\alpha$-approval winner policies.

Theorem 2 shows that for $\alpha\leqslant 1/e$, there always exists a policy that all agents approve, and by Proposition 3 such policies can be found in polynomial time, assuming access to an oracle for volumetric computations. Therefore, the problem of finding an $\alpha$-approval winner is “easy” for $\alpha\in(0,1/e)$. In sharp contrast, for $\alpha=1$ — namely, plurality — Theorem 4 gives a hardness of approximation. The next theorem shows the hardness of computing $\alpha$-approval for $\alpha\in(7/8,1]$ via a reduction from the MAX-2SAT problem. We defer the proof to Appendix A.

Given the above hardness result, to compute the $\alpha$-approval rule, we turn to mixed-integer linear programming (MILP). Algorithm 3 simply creates $n$ binary variables for each agent $i$ indicating whether $i$ $\alpha$-approves the policy, i.e., $F_{i}(J_{i}(\pi))\geqslant\alpha$ which is equivalent to $J_{i}(\pi)\geqslant F^{-1}_{i}(\alpha)$. To encode the expected return requirement for agent $i$ to approve a policy as a linear constraint, we precompute $F^{-1}_{i}(\alpha)$. This can be done by a binary search similar to Algorithm 2. Importantly, Algorithm 3 has one binary variable per agent and only $n$ constraints which is key to its practicability.

Borda count. The Borda count rule also has a natural definition in the continuous setting. In the discrete setting, the Borda score of agent $i$ for alternative $c$ is the number of alternatives $c^{\prime}$ such that $c\succ_{i}c^{\prime}$. In the continuous setting, $F_{i}(J_{i}(\pi))$ indicates the volume of the occupancy polytope to which agent $i$ prefers $\pi$. The Borda count rule then selects a policy among $\operatorname{\arg\max}_{\pi}\sum_{i\in[n]}F_{i}(J_{i}(\pi))$.

The computational complexity of the Borda count rule remains an interesting open question, though we make progress on two fronts.⁴⁴4We suspect the problem to be $\mathbf{NP}$-hard since the objective resembles a summation of “sigmoidal” functions over a convex domain, which is known to be $\mathbf{NP}$-hard [37]. First, we identify a sufficient condition under which we can find an approximate max Borda count policy using convex optimization in polynomial time. Second, similar to Algorithm 3, we present a MILP to approximate the Borda count rule in Algorithm 4.

The first is based on the observation in Section 4 that $F_{i}$ is concave in range $[\mathrm{mode}(f_{i}),\infty)$. We assume that the max Borda count policy $\pi$ appears in the concave portion of all agents, i.e., $J_{i}(\pi)\geqslant\mathrm{mode}(f_{i})$ for all $i\in[n]$. Then, the problem becomes a maximization of the concave objective $\max\sum_{i}F_{i}(\langle d_{\pi},R_{i}\rangle)$ over the convex domain $\{d_{\pi}\in\mathcal{O}\mid\langle d_{\pi},R_{i}\rangle\geqslant\mathrm{mode}(f_{i}),\forall i\in[n]\}$.

Second, Algorithm 4 is a MILP that finds an approximate max Borda count policy. As a pre-processing step, we estimate $F_{i}$ for each agent $i$ separately. We measure $F_{i}$ for the fixed expected return values of $\{\epsilon,2\epsilon,\ldots,1-\epsilon,1\}$. This accounts for $1/\epsilon$ oracle calls to $\mathrm{vol\text{-}comp}$ per agent. Then, for the MILP, we introduce ${1}/{\epsilon}$ binary variables for each agent indicating their $\epsilon$-rounded return levels, i.e., $a_{i,k}=1$ iff $\langle d_{\pi},R_{i}\rangle\geqslant k\epsilon$ for $k\in[1/\epsilon]$. The MILP then searches for an occupancy measure $d_{\pi}\in\mathcal{O}$ with the maximum total Borda score among the $\epsilon$-rounded expected return vectors, where the $\epsilon$-rounded value of $x\in\mathbb{R}_{\geqslant 0}$ is defined as $\lfloor x/\epsilon\rfloor\cdot\epsilon$. Therefore, the MILP finds a policy whose Borda score is at least as high as the maximum Borda score among $\epsilon$-rounded return vectors. This is not necessarily equivalent to a $(1-\epsilon)$ approximation of the max Borda score.

Finally, we make a novel connection between $q$-quantile fairness and Borda count in Theorem 6.

A corollary of Theorems 6 and 2 is that the policy returned by $\epsilon$-max quantile fair algorithm (Algorithm 2) achieves a $(1/e-\epsilon)$ multiplicative approximation of the maximum Borda score.

Environment. We adapt the dynamic attention allocation environment introduced by D’Amour et al. [11]. We aim to monitor several sites and prevent potential incidents, but limited resources prevent us from monitoring all sites at all times; this is inspired by applications such as food inspection and pest control ⁵⁵5The code for the experiments is available at https://github.com/praal/policy-aggregation.. There are $m=5$ warehouses and each can be in 3 different stages: normal ($\mathrm{norm}$), risky ($\mathrm{risk}$) and incident ($\mathrm{inc}$). There are $|\mathcal{S}|=3^{m}$ states containing all possible stages of all warehouses. In each step, we can monitor at most one site, so there are $m+1$ actions, where action $m+1$ is no operation and action $i\leqslant m$ is monitoring warehouse $i$. There are $n$ agents; each agent $i$ has a list $\ell_{i}$ of warehouses that they consider valuable and a reward function $R_{i}$. In each step $t$, $R_{i}(s_{t},a_{t})=-\operatorname{\mathbb{I}}[a_{t}\leqslant m]-\sum_{j\in\ell_{i}}\rho_{i}w_{j}\cdot\operatorname{\mathbb{I}}[s_{t,j}=\mathrm{inc}\land a_{t}\neq j]$, where $w_{j}\in\{100,150,\cdots,250\}$ denotes the penalty of an incident occurring in warehouse $j$, $\rho_{i}$ is the scale of penalties for agent $i$ which is sampled from $\{0.25,0.5,\cdots,n\}$, and $-1$ is the cost of monitoring. In each step, if we monitor warehouse $j$, its stage becomes normal. If not, it changes from $\mathrm{norm}$ to $\mathrm{risk}$ and from $\mathrm{risk}$ to $\mathrm{inc}$ with probabilities $p_{j,\mathrm{risk}}$ and $p_{j,\mathrm{inc}}$, and stays the same otherwise. Probabilities are sampled i.i.d. uniformly from $[0.5,0.8]$. The state transitions $\mathcal{P}$ is the product of the warehouses’ stage transitions.

Rules. We compare the outcomes of policy aggregation with different rules: max-quantile, Borda, $\alpha$-approval ($\alpha=0.9,0.8$), egalitarian (maximize minimum return) and utilitarian (maximize sum of returns). We sample $5\cdot 10^{5}$ random policies based on which we fit a generalized logistic function to estimate the cdf of the expected return distribution $F_{i}$ (Definition 4) for every agent. The policies for $\alpha$-approval voting rules are optimized with respect to maximum utilitarian welfare. The egalitarian rule finds a policy that maximizes the expected return of the worst-off agent, then optimizes for the second worst-off agent, and so on. The implementation details of Borda count are in Appendix B.

Results. In Figure 1, we report the normalized expected return of agents as $\frac{J_{i}(\pi)-\min_{\pi}J_{i}(\pi)}{\max_{\pi^{\prime}}J_{i}(\pi^{\prime})-\min_{\pi^{\prime\prime}}J_{i}(\pi^{\prime\prime})}$ (sorted from lowest to highest) which are averaged over $10$ different environment and agents instances. We observe that the utilitarian and egalitarian rules are sensitive to the different agents’ reward scales and tend to perform unfairly. The utilitarian rule achieves the highest utilitarian welfare by almost ignoring one agent. The egalitarian rule achieves higher return for the worst-off agents compared to the utilitarian rule, but still yields an inequitable outcome. The max-quantile rule tends to return the fairest outcomes with similar normalized returns for the agents. The Borda rule, while not a fair rule by design, tends to find fair outcomes which are slightly worse than the max-quantile rule. The $\alpha$-approval rule with max utilitarian completion tends to the utilitarian rule as $\alpha\to 0$ and to plurality as $\alpha\to 1$. Importantly, although not shown in the plots, the plurality rule ignores almost all agents and performs optimally for a randomly selected agent.

In addition to the fine-grained utility distributions, in Table 1, we report two aggregate measures based on agents’ utilities: (i) the Gini index, a statistical measure of dispersion defined as $\frac{\sum_{i\in N}\sum_{j\in N}|J_{i}(\pi)-J_{j}(\pi)|}{2n\sum_{i\in N}J_{i}(\pi)}$ — where a lower Gini index indicates a more equitable distribution, and (ii) the Nash welfare, defined as the geometric mean of agents’ utilities $\left(\prod_{i\in N}J_{i}(\pi)\right)^{1/n}$ — where a higher Nash welfare is preferable. We observe a similar trend as above, where utilitarian and egalitarian rules perform worse across both metrics. For the other four rules, the Nash welfare scores are comparable in both scenarios, with Borda showing slightly better performance. The Gini index, however, highlights a clearer distinction among the rules, with max-quantile performing better.

We conclude by discussing some of the limitations of our approach. A first potential limitation is computation. When we started our investigation of the policy aggregation problem, we were skeptical that ordinal solutions from social choice could be practically applied. We believe that our results successfully lay this initial concern to rest. However, additional algorithmic advances are needed to scale our approach beyond thousands of agents, states, and actions. Additionally, an interesting future direction is to apply these rules within continuous state or action spaces, as well as in online reinforcement learning setting where the environment remains unknown.

A second limitation is the possibility of strategic behavior. The Gibbard-Satterthwaite Theorem [16, 33] precludes the existence of “reasonable” voting rules that are strategyproof, in the sense that agents cannot gain from misreporting their ordinal preferences; we conjecture that a similar result holds for policy aggregation in our framework. However, if reward functions are obtained through inverse reinforcement learning, successful manipulation would be difficult: an agent would have to act in a way that the learned reward function induces ordinal (volumetric) preferences leading to a higher-return aggregate stochastic policy. This separation between the actions taken by an agent and the preferences they induce would likely alleviate the theoretical susceptibility of our methods to strategic behavior.

We gratefully acknowledge funding from the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Canada CIFAR AI Chairs Program (Vector Institute). This work was also partially supported by the Cooperative AI Foundation; by the National Science Foundation under grants IIS-2147187, IIS-2229881 and CCF-2007080; and by the Office of Naval Research under grants N00014-20-1-2488 and N00014-24-1-2704.