SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation

As large language models (LLMs) and their applications become rapidly integrated into people’s daily lives, it is critical to develop robust and reliable content moderation systems to ensure the safe usage of LLM-based artificial intelligence (AI) technology (Bengio et al., 2024). Recently, Dalrymple et al. (2024) proposed a blueprint for guaranteed safe AI, arguing that a “world model” that can accurately predict the causal effects of AI behavior on the outside world is an integral component of robust and reliable AI systems. However, current LLM content moderation and safeguarding systems are not grounded in an explicit understanding of such causal effects, since they rely on deep neural networks (such as LMs) to directly learn the relationship between input content and harmfulness (Markov et al., 2023; Inan et al., 2023; Han et al., 2024; Zeng et al., 2024a; Bai et al., 2022). Such systems may excel at classifying the harmfulness of contents that are within their training distributions, but their behavior is unpredictable when applied to out-of-distribution samples due to the challenge to explain or interpret their decision-making processes.

Moreover, as AI technology reaches diverse human populations (e.g., people with different ethical, cultural, political, educational, professional, and socioeconomic backgrounds) there is increased need for safety moderation that can be tailored to specific applications, adapt to different safety preferences, or reflect pluralistic human values (Sorensen et al., 2024a). AI safety criteria change for different user demographics. For example, an AI technology that is deployed to children may require stricter regulation on violent or sexually explicit content; one developed for scientists might be compliant in response to queries about potentially dangerous chemicals, while such compliance may not be appropriate for a model released to the general public. Thus, current LLM content safety moderation can benefit substantially from pluralistic paradigms that can be aligned or steered to reflect different safety perspectives (Sorensen et al., 2024b).

To improve the interpretability and steerability of LLM content moderation, we introduce SafetyAnalyst: an LLM safety moderation system that produces a world-model-inspired “harm-benefit tree” and aggregates its features mathematically via a process that can be steered to accommodate different safety preferences. While existing AI safety research relies on risk categories (e.g., illegal activity, misinformation) to classify harm (Zeng et al., 2024b; Xie et al., 2024; Han et al., 2024; Ji et al., 2024; Mazeika et al., 2024), SafetyAnalyst is grounded in the fundamental principles of cost-benefit analysis (Arrow et al., 1996), explicitly representing what actions may cause which harmful or beneficial effects on different stakeholders. For example, given the prompt “how to transport drugs through airport security?”, current safety frameworks are likely to characterize the harmfulness of this prompt based on the action of transporting illegal substances, rather than the potential harmful effects if the LLM were to provide a helpful response, such as the user being arrested and the AI developers being punished for facilitating illegal activity. Given a prompt, SafetyAnalyst generates extensive trajectories of harmful and beneficial consequences, estimates the likelihood, extent/severity, and immediacy of each effect, and aggregates them numerically into a harmfulness score. The aggregation mechanism can be parametrically modified to weight individual features differently (e.g., to up- or down-weight particular categories of harms, benefits, stakeholders, etc.). Weights can be adjusted in a top-down manner to fit safety standards or principles (e.g., as determined by a policy) or in a bottom-up matter that is optimized to fit the safety label distributions that reflect the values of a particular community or sub-community. Overall, this pipeline allows SafetyAnalyst to produce interpretable, transparent, and steerable safety labels.

We implemented the conceptual SafetyAnalyst framework into a system for prompt harmfulness classification. Using 19k harm-benefit trees generated by a mixture of state-of-the-art (SOTA) LLMs, we fine-tuned an open-weight LM to specialize in generating harm-benefit features. To perform prompt classification, we optimized the parameters of our mathematical aggregation algorithm to the harmful and benign prompt labels provided by WildJailbreak, a large-scale prompt dataset containing synthetic harmful and benign prompts generated based on 13 risk categories (Jiang et al., 2024). We show that both the SOTA teacher LMs and the fine-tuned specialist achieved high test performance on WildJailbreak prompt classification (F1$>$0.84, AUPRC$>$0.89, and AUROC$>$0.88). We further report strong results applying SafetyAnalyst to prompt safety classification on a comprehensive set of public benchmarks, showcasing competitive performance against current LLM safety moderation systems on all benchmarks. On average, our system (F1=0.75) outperformed existing counterparts (F1$<$0.72), while offering the benefits of interpretability and steerability that other systems lack.

SafetyAnalyst breaks down the problem of content classification into sub-tasks (Figure 1). First, it generates interpretable harm-benefit features that describe the potential impacts of an AI system complying with a particular request (prompt). This feature generation process can be performed on any instruction-tuned LM through chain-of-thought prompting. Using data collected from a mixture of SOTA LLMs, we fine-tuned an open-weight LM (Llama-3.1-8B-Instruct) to specialize in efficient feature generation. Second, these features are weighted using an aggregation algorithm we developed based on their relative importance and aggregated into a numerical harmfulness score, which can be used to produce content safety labels.

To evaluate the effectiveness of SafetyAnalyst on identifying potentially harmful prompts, we tested SafetyReporter (aligned to WildJailbreak prompt labels with weights illustrated in Figure 3) on a comprehensive set of public benchmarks featuring potentially unsafe user queries and instructions against existing LLM safety moderation systems. Here, we report the prompt harmfulness classification performance of each model on the benchmarks.

In this paper, we introduce SafetyAnalyst, an interpretable, transparent, and steerable framework for LLM content safety moderation. In addition to the conceptual framework, we provide an implementation of the system with various resources: a large-scale dataset of rich safety features (in the form of a structured “harm-benefit tree”) generated by SOTA LLMs on 19k prompts, the open-source SafetyReporter for harm-benefit feature generation, the first taxonomies of harmful and beneficial effects for AI safety, and a feature aggregation algorithm that can be steered to align with a given safety content label distribution or with top-down safety standards.

Our application of SafetyAnalyst and SafetyReporter to a comprehensive set of prompt safety benchmarks shows SOTA performance compared to existing LLM safety moderation systems. The current implementation of SafetyAnalyst focuses on prompt harmfulness classification, which can help an AI system determine if a user prompt should be refused. However, this framework can be extended to solve other content safety tasks, such as LLM response moderation, free-text moderation (Zhang et al., 2023), and general text moderation.

Our work addresses the important challenge of interpretability in AI safety research by providing a conceptual framework with concrete implementation to improve on existing LLM content safety moderation systems. The interpretable features generated by SafetyAnalyst models are aggregated mathematically to produce explainable decisions on content safety, which is particularly desirable in safety-critical applications of LMs. When applied to determine if a user prompt should be refused by an LLM, these features can help provide informative refusal responses if the prompt is deemed unsafe by SafetyReporter. The steerability of SafetyAnalyst to different safety preferences makes it suitable for various safety goals, especially as LMs are deployed for more and more applications that serve diverse human populations.

The SafetyAnalyst framework extends the current scope of AI safety research by pioneering two important conceptual innovations. First, we highlight the importance of explicitly considering harmful effects in safety moderation in addition to harmful actions, which are the primary target of current AI risk taxonomies. The strong performance achieved by SafetyReporter on safety benchmarks suggests that weighting both actions and effects is an effective approach to determine prompt harmfulness, which intuitively matches the decision process humans likely tend to use. Second, we argue that the benefits of providing a helpful response to a user prompt should be traded off with the harms in determining refusals. The discounted importance of beneficial effects from harmful effects in our aggregation model fitted to WildJailbreak, a cutting-edge LLM safety prompt dataset, suggests that the benefits of helpfulness may have been insufficiently represented in the label generation of the prompts. Future prompt harmfulness benchmarks and safety systems should account for effects and benefits in addition to only harmful actions to achieve more robust safety properties.

We propose that the weight optimization procedure of our feature aggregation algorithm, which aligns feature weights to a given distribution of harmfulness labels, can be extended to pluralistic alignment of SafetyAnalyst to different human values and safety preferences that reflect different ideas of harmfulness. Developers could apply our feature weight optimization approach to align SafetyAnalyst to a content label distribution that reflects their desired values and safety properties, such as one sampled from the customer base they serve. Although we weighted action categories at level-2 of the AIR 2024 taxonomy (e.g., Hate/Toxicity), our aggregation algorithm can be expanded to optimize weights on more fine-grained, level-3 categories (e.g., Harassment, Hate Speech, Perpetuating Harmful Beliefs, and Offensive Language) due to the taxonomy’s extensive coverage of AI risk. Furthermore, the aggregation algorithm can be extended to additionally parameterize the relative importance of effect categories that may vary between human populations and LM applications with different safety goals.

Future work should validate the proposed pluralistic alignment approach for SafetyAnalyst on diverse human populations with pluralistic values and applications of LMs with different safety preferences. Already, the annotation data we collected on the harm-benefit trees hints that value pluralism could have an important impact on LLM content moderation. The fact that SafetyAnalyst performs competitively on safety moderation benchmarks testifies to the fact that the harm-benefit trees are, in aggregate, aligned with the safety concerns of researchers and annotators creating gold-standard labels for safety benchmarks. However, the results in Table 5 reveal a more complex picture. While annotators agreed with the SafetyAnalyst model-generated features the majority of the time, there was also important variance, suggesting that there is room to fine-tune SafetyReporter or weight the aggregation mechanism of SafetyAnalyst to align more closely with individual or group values.