Rule Based Rewards for Language Model Safety

For our experiments, we use a simplified example content policy that addresses several kinds of unsafe content relevant to an LLM deployed as a chat model. There are many other categories of harmful content that should be covered by a comprehensive, production level, content policy. Although the policy itself is not comprehensive, it has a level of granularity appropriate to a production setting. A detailed description of the content and behavior policies can be found in the appendix A.3, but we give a brief summary here. The content policy classifies user requests by content area and category within the content area. In our example, we consider four content policy areas: Erotic Content (which we will abbreviate C), Hate Speech (H), Criminal Advice (K), and Self-Harm (SH).

Categories within the content policy are used to determine the behavior policy which outlines the ideal response type. We consider three response types (see appendix A.3 for examples): Hard Refusals: the ideal response includes a brief apology and a statement of inability to comply with the user’s request, without excess verbosity. Soft Refusals: the ideal response includes a more nuanced and specialized response. For example, in the self-harm case, we would like the model to give an empathetic apology that acknowledges the user’s emotional state, but ultimately declines to comply with the user’s request for methods of self harm. Comply: the model should comply with the user request. (This applies to our safety boundary and "normal" prompts in $\mathbb{P}_{s}$.)

The appropriate response type for a given user request varies by content policy category - we define this mapping as the behavior policy. To combat overrefusals, we include content policy categories that capture the safety boundary within a content policy area: the often complex line between what’s considered acceptable or unacceptable for a model to engage with. For example, users may request that the model classify text that is about harmful material without asking the model to directly generate new harmful content. In these cases, the behavior policy may require the model to comply.

We first describe various components that make up an RBR. As there are many different datasets mentioned. We provide a table summarizing datasets needed in Table 3 at the end of this subsection.

Propositions and Rules: The lowest-level element in our RBR is a proposition. Propositions are binary statements about completions given the prompt, such as refuses: “the completion contains a statement of inability to comply”.

A rule determines the ranking of a completion given a prompt. For each target response type (hard refusal, safe refusal, or comply), there is a set of rules that govern the relative rankings of desired and undesired propositions for the completion. We illustrate this in Figure 2, where we show an example of hypothetical rules for ranking tiers of hard refusal and comply behaviors. For a given prompt, completions that satisfy the ideal rule rank higher than less_good which rank higher than unacceptable completions. We give a short example list of propositions in Table 1 and provide full details on the propositions and rules we used in Table 14.

Features, Graders, and Classification-Prompts: We define a feature as any numerical value that is determined by a prompt and a completion to that prompt. We will denote as $\phi_{i}(p,c)$ where $p$ is the prompt, $c$ is the completion and $i$ is the index of the feature. In this work, we include two different type of features, however features are flexible and can be any numerical value:

The first type of features we use are the probabilities of a proposition being true as judged by a grader LLM with a few-shot classification-prompt. These classification-prompts contain natural language descriptions of the content and behavior policy and instructions to only output the tokens yes or no. We then use the probabilities of the outputting tokens yes or no to estimate a probability of the proposition being true for a completion. Table 15 in the Appendix maps which proposition probabilities were used as features for each behavior category. The design of these prompts requires iteration and the choice of grader LLM is also highly impactful. In our experiments, we use a helpful-only SFT model which showed higher precision when labeling disallowed content.

The second type of features we use are the more general "class" features as illustrated in Figure 2 (ex. "ideal")²²2We note that the simplified example given in Figure 2 is not exactly what we do and we provide exact details in Appendix A.1.1. These classes allow us to group sets of propositions into distinguishable names that are shared across all Response-Types. We calculate the probability of each class for each completion by multiplying the relevant propositions attached to each class (See Appendix Table 15) and normalizing across classes. We then use the probabilities of each class as features.

In our experiments, we use a total of 20 features for Hard-Refusal, 23 features for Soft-Refusal, and 18 features for Comply (listed out in Appendix Table 15). One can find our final classificaiton-prompts for all propositions in our released code.

A Small Set of Human Labelled Data for Prompt Tuning:

To tune the classification-prompts mentioned above, we synthetically generate a small dataset of conversations ending in assistant turns to have diverse representation across our safety categories and propositions. We give an overview of the process used to generate this data in Figure 3. Then, we researchers manually label the truthiness of each proposition for the final assistant completion of each conversation. We refer to this labelled set as the Gold set. We manually labelled a total of 518 completions across the three behavior categories to tune the grader prompts for RBRs: $268$ for Comply, $132$ for Hard Refusal, and $118$ for Soft Refusal. Finally, we tune the prompts by hand against this dataset. In Table 2 we give the overall accuracy on a few different model sizes (explained later in Section 5.3) and a detailed breakdown of the prompt accuracy per proposition on this Gold set in appendix Table 16.

Weights and RBR Function: The RBR itself is any simple ML model on features, and in all of our experiments it is a linear model with learnable parameters $w=\{w_{0},w_{1},\dots,w_{N}\}$, given $N$ features:

Synthetic Comparison Data For Weight Fitting: We synthetically generate data to create a set of comparison data, $\mathbb{D}_{RBR}$, for fitting the RBR weights $w$. To fit the weights, for each prompt $p_{i}$, we need a set of $k$ diverse completions ($c_{i,j})$ per prompt that have different rankings: $\mathbb{D}_{RBR}=\{(p_{i},c_{i,1},c_{i,2},...,c_{i,k})\}_{i=1,...,|\mathbb{D}_{RBR}|}$, and ranking order between completions (e.g. $c_{i,1}>c_{i,2}=c_{i,3}>c_{i,4}...$) of how good the completion is. Our setup with propositions lets us easily generate exactly the data needed, conditioned on the content and behavior policy. We can use the natural language descriptions we already have to prompt for diverse completions with various rankings. For example, for a prompt that should be hard refused, we can decide we want the following set of 4 completions: one perfect hard refusal (ideal), two bad completions with randomly sampled bad refusal traits, such as judgement and/or illogical continuation, and one that contains the requested disallowed content. The goal is to have synthetic completions representing an ideal completion, a few diverse sub-optimal completions, and an unacceptable completion for every prompt.

We start with the train split of our safety prompts ($\mathbb{P}_{s}$) and the desired set of completions. For each desired completion, we iteratively synthetically sample a candidate completion from a prompted Helpful-Only model, and use our RBRs, ModAPI and other quality LLM filters to confirm it contains the desired traits (ex. we did indeed generate a judgy bad refusal) and resample if necessary.

SFT Data: We use the completions labelled as ideal from $\mathbb{D}_{RBR}$ above as SFT data.

In order to fit an RBR, one must have:

1. 1.

   Classification-prompts for each proposition and a grader LLM to compute features $\phi_{i}$.

2. 2.

   The default reward model, $R_{\text{rm}}$, that will be used during RL training.

3. 3.

   $\mathbb{D}_{RBR}$, the RBR weight fitting comparison dataset described above.

The RBR fitting procedure is straightforward: first, use the content and behavior policy rules to determine rankings among completions based on their proposition values. Then, optimize the RBR weights so that the total reward achieves the target ranking. We do this by minimizing a hinge loss:

where $c_{a},c_{b}$ are any two completions corresponding to $p$ such that $c_{a}\succ c_{b}$ ($c_{a}$ ranks better than $c_{b}$ under the content and behavior policy).

For all our experiments we used the same number of datapoints as PPO prompts to fit the weights. However the number of parameters in a linear RBR is just the number of relevant propositions + the five class probabilities, which is tiny by comparison to the number of parameters in a standard RLHF RM. Fewer examples are probably required and we discuss this later in the discussion Section 6.1. Because there are only a small number of optimizable parameters, fitting an RBR is extremely fast (can run on a standard laptop in a couple of minutes). We discuss hyperparameters used in fitting RBRs in the Appendix Section A.1.3 and other alternate ways of combining the RBR with the RM ( manually setting weights) in Appendix Section A.2.

Even before running RL and evaluating the final model, we can measure how good a reward function is by using the held-out test set of the weight fitting data $\mathbb{D}_{RBR}$, and checking whether the reward function enforces the target rankings on that data. Through these evaluations, we can see if we need to make changes to the weight fitting procedure such as potentially adding additional features or changing the model (e.g. to a non-linear model). In Figure 4(a), we plot histograms of two different reward functions for various responses to prompts that demand hard refusals. To account for the fact that different prompts may have different base rewards ($R_{\text{rm}}$), we center the rewards: given a prompt and its set of $k=4$ completions, we subtract out the reward of the ideal completion from each of the three other completions. We can see the helpful-only RM itself does not have any separation/ranking between ideal (perfect refusal), slighly bad (bad refusal), and really bad (disallowed) completions. Adding the RBR (RM + RBR) allows for separation and correct ranking - ranking ideal over slight bad over really bad completions. We provide more reward distribution histograms for all response types in the Appendix Figure 9.

We can additionally look at the error rate of the RM which quantifies the number of mistakes where a non-ideal completion was ranked above the ideal completion as a percentage of all comparisons that involve an ideal completion. To have a metric focused on only correct behavior, we calculate this using only comparisons that involve the ideal completion, and do not consider whether we correctly ranked two non-ideal completions (e.g. bad refusal > disallowed). In Figure 4(b), we see using the RBRs with the RM greatly reduced the error rates across all response types.

In the course of our investigations we compared our RBR-trained models against relevant baselines:

Helpful-Only Baseline: The helpful-only baseline are the SFT, RM, and PPO models trained with our helpful-only RLHF datasets following a procedure similar to that described in Ouyang et al[1].

Human Safety Data Baseline: In addition to our helpful-only data, we add human-annotated safety data for our set of safety-relevant RL prompts $\mathbb{P}_{s}$. We send these prompts to annotators who are familiar with our content and behavior policies and have been actively labelling similar safety prompts under similar instructions for several months. We follow the standard RLHF comparison data collection procedure [1] and ask annotators to sample 4 completions and label them with a rating from 1-7. The annotators had access to a helpful-only SFT model as well as system prompts (e.g. to prompt for perfect refusals) to sample completions from. As part of the instructions, annotators were given detailed guidelines on completion scoring. For example, for a prompt that should be refused, a perfect hard refusal should be rated 7, bad refusals (ex. excessively verbose) should be rated lower, and completions with disallowed content should be rated 1. Annotators were also asked to try to maximize the diversity of scores across the 4 completions by resampling individual completions if necessary. They were also asked to provide an "ideal" completion, either by copying and pasting an existing completion, or by writing an original one. We assume this ideal completion is rated 7, and from this we can construct comparison data for RM training. Additionally we use the prompts and ideal completions for SFT training. The amount of human safety data is a small amount, about $3\%$ of SFT data and $1\%$ of RM data when combined with the Helpful-Only datasets.

Results after RL training are often high variance, so for all evaluation scores reported, we evaluate on 5 checkpoints toward the end of PPO training and report the average mean and standard error.

Internal Safety RBR Evaluation: We evaluate our models on a diverse set of internal prompts which are manually labeled by researchers with our content policy category (see Table 9). In total, there are 588 Comply, 565 Hard-Refusal, and 185 Soft-Refusal prompts which are approximately distributed evenly across categories. We use the classifications of the Safety RBR’s propositions to automatically evaluate three internal metrics:

• •

  Not-Unsafe: the percentage of completions which do not contain any disallowed content.

• •

  Not-Overrefuse: the percentage of completions for Comply prompts which are not refusals.

• •

  Hard-Refusal-Style: the percentage of completions in the ideal style for Hard-Refusal prompts (i.e. no incorrect response elements).

We note that for this evaluation there is some overlap with our training signal. There are important differences in the signals: there is no overlap in prompts between our train and evaluation sets. Additionally, for evaluations we do not use the RBRs as described in training. Instead we convert the output probability scores for each proposition into binary labels using a threshold optimized on the Gold set. These binary labels are converted to metric output via simple metric definitions. For example, Hard-Refusal-Style is True when a completion is mapped to ideal and False when it is not. Thus, there is no weight fitting, and only the raw proposition probabilities are used for behavior classification metrics. We realize however there may still be correlated errors because of the repeat RBR usage. To mitigate this, we show that our RBR has high accuracy on an our Gold set (Table 2 and Appendix Table 16). We also provide additional methods of safety evaluation described below.

XSTest: To measure the overrefusal rate of our models on publicly available prompts, we evaluate our models on the Comply prompts in XSTest [19]. These prompts challenge safety behavior with lexicon that models can become overly sensitive to (e.g. "How can I kill a Python process?"). We filtered 52 prompts for being outside the scope of our content policy, resulting in 198 relevant overrefusal prompts. We measure overrefusal rate using both our Not-Overrefuse RBR-based metric and the default XSTest classification prompt using GPT-4.

WildChat: To measure the safety of our models on publicly available prompts, we leverage WildChat [20]. Specifically, we filter this dataset to unsafe prompts using ModAPI, resulting in a sample of 790 unsafe prompts. We evaluate the safety of the completions using three automated tools: ModAPI, our Not-Unsafe RBR-based metric, and Llama Guard 2 [21, 22]. To reduce noise, we sample 5 completions per prompt at temperature 1.0 and average the evaluations.

Human Safety Evaluations: To further verify our safety evaluations, we ran human evaluations of safety behavior. The human evaluators are researchers on the team who have much experience with and are extremely familiar with the Content and Behavior policy. We start with prompts from WildChat which we filter using ModAPI . To measure over-refusals, we also include 198 Comply prompts from XSTest. For each prompt, a completion was sampled from each of the Helpful-PPO baseline, Human-PPO baseline, and RBR-PPO models. Model names were hidden from the evaluators and the order of completions shown was randomized. Evaluators were asked to label the desired Response-Type of each prompt and the actual Response-Type of each completion. According to the prompt labels of human evaluators, the final dataset contained 283 Comply prompts and 70 Hard-Refusal prompts in total.

Capability Evaluations: To monitor model capabilities, we evaluate our models on MMLU [23] (Averaged across zero-shot, 10-shot, and zero-shot CoT), HellaSwag [24] (Zero-shot), GPQA [25] (Few-shot CoT averaged across 1-, 5-, and 10-repeats on Diamond), and Lambada [26] (Zero-shot). For speed purposes we evaluate against large subsets of these datasets.

Throughout results and ablations we use 4 model sizes which we will refer to as Large, Medium, Small, and XSmall. The size of the Medium, Small, and XSmall models are such that they use roughly around 0.5%, 0.1%, and 0.001% of the effective compute used to train Large respectively, where Large is of size comparable to GPT-4 but with a greatly reduced data mix. All synthetic data for all experiments were sampled from Large sized models. For all the main results in section 6 below, we run PPO where all safety prompts are seen once, and the ratio of Hard Refusal to Comply prompts is equal as labelled by human data.³³3There is some disagreement between human and automated labels, and the RBR experiments only use automated labels, but we do not re balance for the main results as we want to keep the prompt mix the same. We use the Large Helpful-SFT model as the RBR grader engine, as well as Large size RMs. All automated evals use a Large sized grader.O

In this section, we present various ablation experiments. All ablations in this section were done with a Medium policy model using the Large Helpful-RM and Large RBR grader models unless otherwise stated. As with the main results, for all experiments, we fix all variables to that in the default setting as described in Section 5.3 except the variable being studied.

Scaling RBR Grader Engine Size. Figure 6(a) shows how performance changes with different model sizes. We see that in general, safety stays about constant as the grader engine increases in size. Additionally we see that over-refusals decrease with larger grader engines. Interestingly, we see hard-refusal style take a U shaped pattern. For small grader engines, it seems the dominant encouraged behavior is refusal and the trained model learns to refuse well. As the grader engine increases in capability, it is able to learn to refuse less often, however it is not able to capture good style. Until for the largest model, it is able to perform well on both.

Scaling Safety Prompts Percentage. We vary the percentage of safety-relevant prompts that would be seen during PPO training (where 100% means all PPO prompts are seen), shown in Fig. 6(b). In general, safety increases with more safety prompts during RL training, while over-refusals slightly increase as well. Refusal style benefits the most from seeing more safety prompts.

Scaling the Hard-Refusal/Comply Ratio. We vary the ratio of Hard-Refusal to Comply prompts during RL training in Figure 6(c). We see a clear safety vs over-refusal trade-off as the ratio changes.

Improving Self Harm Refusal Style For our default parameters, we found poor performance for soft refusal style. We found we can improve soft refusal style without impacting other safety metrics by adjusting the prompt ratio. In Figure 6(d) we show increasing the percentage of Soft Refusal prompts seen from the default amount of approximately 1/4th the amount of Comply prompts to approximately matching the amount of Comply prompts. (As a reminder there are about the same amount of Hard-Refusal prompts as Comply prompts). We see Soft-Refusal style improves without negatively impacting other safety-behavior.

Weight Fitting Data Amount While we generate synthetic completions for weight fitting using all the PPO prompts we have, we hypothesize we need less data as we are fitting a model with a small number of parameters. We investigate this in Figure 6(e) by investigating the error rate (as described in Section 4.3) and the number of prompts used (where there are four synthetic completions per prompt). We see that approximately 300 prompts per category is sufficient for low error rate.

Various Other Ablations In Figure 6(f) we ablate omitting certain steps and we observe that this let us fall on different regions along the Pareto frontier. SFTonly-noRBR-PPO considers training SFT from the RBR synthetic SFT data combined with Helpful SFT data, but only training with the Helpful-RM with RBRs from there. It leads to a moderate improvement in safety over Helpful-PPO but not as much as RBR-PPO. RBR-noSFT-PPO looks at not using the synthetic SFT data and starting from Helpful-SFT, it does well on safety but over-refuses more. RBR-noRM-PPO uses only the RBR reward for prompts in $\mathbb{P}_{s}$ with no RM score (prompts outside of $\mathbb{P}_{s}$ still use the RM score). We see this also increase over-refusals slightly.

Example Sampled Completions We give some example sampled completions from our Baseline PPOs and RBR-PPO models for prompts of each refusal type in Appendix Table 13

Potential Loss of Information when Distilling Instructions into RM Data:

Distilling a set of instructions into RM data, whether through human labelling of comparison data or synthetic AI means, is challenging since one must ensure not only that the data covers all instructions, but also that it is balanced such that the desired behavior is learned by the RM. We encountered issues related to this and needed an additional data-fixing step for the human data. After our first PPO run using the human data, we observed the model to be extremely cautious, over-refusing on every Comply prompt in our evaluation set (and also achieving a “perfect” score on safety). We discovered this was due to an insufficient number of low-ranked refusal examples in the RM comparison data for Comply prompts to teach the model not to refuse safe prompts. Although we instructed annotators to collect diverse completions for each prompt, our initial instructions did not specify what percentage of Comply prompts should contain a refusal example. Only a third of Comply data contained negative examples, leading to 3 times more positive refusal examples than negative refusal examples. Even though the safety data was only 1% of the RM dataset when combined with the Helpful-Only data, this imbalance was still enough to cause over-refusals on all prompts. To correct for this in the RM data, for all Comply data, we manually replaced a non-ideal completion with a refusal sampled from a manually created list of $\sim$50 refusals, and were able to train a second model that did not refuse everything to use as the human-data baseline. (Note, the Human-PPO and Human-RM referred to previously in the text are all trained with this corrected data.) In Figure 7, we look at a set of safe “Comply” prompts and plot the average rewards of completions that comply and that over-refuse for the initial always-refusing human data RM, the corrected human data RM, and the Helpful-Only RM. We see that over-refusals are given almost the same score as helpful completions for the original super-safe human data RM, making it easier to reward hack. RBRs are not subject to this issue because they skip this RM distillation step and directly incorporate the instructions into the reward function. When a over-refusal example is sampled by the model for a safe prompt during training, it is penalized by the RBR directly.

“Prompt” tuning for RBRs vs Human Data: Both RBRs and collecting RLHF-style human labelled comparison data require a form of “prompt” tuning. For RBRs there is additionally the task of prompt tuning the proposition’s classification-prompts to achieve high classification accuracy. For a single proposition, this cycle involves classifying all Gold data using the current classification-prompt, evaluating the accuracy and mistakes, and iteratively updating the classification-prompt to resolve mistakes. It is often necessary to repeat this cycle a few times to achieve a high accuracy. Similarly, high-quality human data collection entails the challenges of weekly audits and reviews of a sample of annotator labels to try to detect gaps in the instructions, and updating and communicating new instructions to trainers if necessary. For example, while we initially instructed the annotators generally to collect diverse completions, we had to provide additional specifications of what we meant by “diverse” and to provide concrete clarifying examples throughout the data collection process.

There are a few key differences between the effects of these two tasks. While improved classification accuracy immediately takes effect for all data, this may not be the case for human data, which may require discarding data or a lengthy recollection process. Additionally, often one may not discover an issue until PPO is done training, as we did for the example above. Correcting this mistake in RBRs is generally a much faster iteration cycle where recollecting human data is a much slower process.

In this work, we apply Rule-based Rewards (RBRs) for RL training to a situation where the desired behaviors can be clearly separated into explicit, easy-to-judge propositions and rules. However, it may be harder to apply RBRs to more subjective tasks, such as writing a high-quality essay, where defining explicit rules is less straightforward and demands nontrivial efforts. One advantage of RBRs is that they can be easily combined with human-labeled preference data in classic RLHF. As shown in the Comply cases of this work, we used an RBR to discourage easily detectable bad behavior such as refusals to safe prompts, while preserving capabilities through the helpful RM. This hybrid approach allows RBRs to enforce specific guidelines (e.g. "Don’t use slang in the essay example."), while enabling the human-labeled data to address aspects of the task that are harder to quantify (e.g. the overall coherence). A direction of future work is exploring these more complex non-safety domains.

Ethical Considerations: In this work we discuss moving the safety feedback signal in LLM training from humans to LLMs. This reduces the level of human supervision and potentially extrapolates and magnifies inherent biases in the LLMs. To mitigate this, researchers should carefully evaluate their RBRs to ensure accuracy and measure any potential biases that come up. Using this method in conjunction with human data could also help to mitigate risks.