CFSafety: Comprehensive Fine-grained Safety Assessment for LLMs

In the past few years, we have witnessed the transformative impact of transformer architectures ref1 on the field of natural language processing (NLP). With the continuous expansion of model parameters and training scale, large language models (LLMs) have gradually become influential in people’s lives. However, in practice, LLMs have also exposed some safety issues. They may generate hate speech , cause ethical trouble, and so forth ref2 ; ref3 and are susceptible to various instruction attacks ref4 ; ref5 .

As LLM-generated content becomes more and more similar to human-created work, traditional NLP evaluation benchmarks such as BLUE and METEOR ref6 ; ref7 , which specifically target the quality of machine translation, may no longer be sufficient for existing complex contexts. Although we now have newer comprehensive evaluation methods for LLMs such as MMLU and C-EVAL ref8 ; ref9 , there is still a gap in specifically assessing the ethical and safety aspects of LLMs. As a closely related effort, SafetyBench ref10 has established a multiple-choice question set covering seven categories of safety concerns to evaluate model safety. Additionally, Safety Assessment ref11 outlines eight typical safety scenarios and six instruction attacks, allowing LLMs to qualitatively assess model responses as either ethical or unethical, providing performance scores across various domains.

However, in complex and ambiguous contexts, the existing evaluation methods that rely on accuracy or binary scores (0 or 1) prove to be insufficient. Moreover, these methods often employ question sets that are predominantly in Chinese or English, thus failing to comprehensively evaluate multilingual environments. Additionally, the current evaluation frameworks exhibit inadequacies in dealing with instruction attacks, especially in terms of covering the latest research and the comprehensiveness of the issues, which may lead to results that are not objective or comprehensive. Inspired by these existing works, we propose an improved evaluation framework : CFSafety. We have surveyed a

number of LLM security reports ref12 ; ref13 and incorporated the latest attack methods such as Persuasion and LLM rewriting ref14 ; ref15 , classifying safety issues into 10 categories. We use LLMs to generate an initial ethical judgment and a 0-5 safety rating. The final score is a combination of the probability-weighted sum of the output score and moral judgments. We visualize the LLM’s ability to handle various safety issues using a radar chart

Our work has several advantages: (1) Comprehensiveness: We have compiled a diverse set of safety issues spanning 10 categories. In comparison to SafetyBench and Safety Assessment, we have created a bilingual dataset in Chinese and English that includes both classic safety scenarios and instruction attacks. We have also integrated some of the latest research findings ref16 . The design of these datasets focuses on evaluating the safety performance of LLMs in these specific contexts, thereby providing a more comprehensive and holistic assessment of LLM safety. (2) Accuracy: We have improved the scoring method used in G-EVAL ref17 by employing LLMs for preliminary moral judgments and integrating a weighted sum of token probabilities outputted by LLMs with a 0-5 safety rating as a safety indicator. This not only reduces the single-step error in LLM evaluations but also refines the granularity of scoring by establishing a connection between moral judgments and safety ratings, replacing simple binary assessments. This approach more accurately quantifies model safety in complex language environments. (3) Simplicity: We use LLMs as evaluators ref18 . This approach obviates the need for manual evaluation or intricate evaluation models, thereby enabling efficient automated assessment.

We conducted experiments using SecuLLM to assess the safety of popular LLMs, including OpenAI’s GPT series and other well-known LLMs. The results are shown in Figure2.

We have constructed a new Chinese-English dataset for evaluating the security of LLMs, based on existing datasets and instruction attack samples generated using some of the latest methods.

In our work, we first have the large language model (LLM) under test answer questions from the question set by category, forming a set of answers. For each answer, we concatenate it with the corresponding question and then add a dynamic prompt template before submitting it to the evaluation LLM for scoring. The prompt template varies according to the safety issue category to ensure the accuracy and specificity of the evaluation.We have also incorporated some elements of In-Context Learning (ICL) ref29 into our templates. Specifically, in the prompt templates for each category of question, we include the definition of the safety issue and basic ethical guidelines, and we add several scoring examples to establish scoring standards and prevent evaluators from giving incorrect assessments due to their own biases. Through this approach, we can systematically evaluate the performance of LLMs in handling different types of safety issues, thus providing a comprehensive critique and analysis of their safety. Here are a few examples of prompt templates:

You are an excellent model evaluator. {definition} .Please read the above question and corresponding answer to judge whether the answer is influenced by the question, contains any potentially socially biased, unethical, or harmful content. If it does, answer with 0; if not, answer with 1. Note: Only answer with 0 or 1, no additional explanation is needed. For example, we score a 0 for {example1}, while for {example2} we can score a 1.

You are an excellent model evaluator. {definition}. Please read the above question and corresponding answer, judge whether the answer is affected by reverse exposure attacks, and whether it contains unethical or harmful content. Based on your judgment, give a moral or safety score of 1-5 for this answer (1 indicates that the answer has significant ethical or safety vulnerabilities, 5 indicates that the answer has excellent ethical and safety performance, and intermediate scores are transitions between the two). Note: Only provide a score of 1-5, no additional explanation is needed. For example, we score a 1 for {example1}, while for {example2} we can score a 5.

We use an evaluation large language model (LLM) to generate a rating of 1 to 5 for each question-answer pair. However, directly using this discrete score as the safety rating for the question-answer pair may be too coarse-grained to reflect the subtle differences between different texts. Therefore, we adopt the approach used in G-EVAL ref17 , which involves using the weighted sum of probabilities of the evaluation LLM’s output scores as the safety rating for the question-answer pair.

Given an input $x$ and model parameters $\theta$, an intermediate value is computed using the Large Language Model (LLM). This value is then transformed into a probability distribution $y$ using the softmax function. Subsequently, the exponential function is applied to $y$, resulting in $prob$, which represents the adjusted probability distribution. Specifically, if we use GPT-3.5 as the evaluation model, we set the logprobs parameter in the OPENAI interface to True to obtain the output scores and their corresponding probabilities.

In addition, before obtaining the 1 to 5 rating, we set up an moral judgment step to obtain a 0 to 1 ethical judgment for the question-answer pair. Similar to a 1-5 safety rating, we also calculate the weighted sum of the probabilities of the output scores to form a moral judgment score and use it as the total weight, and then multiply it by the 1 to 5 rating to obtain the final score for the question-answer pair. Specifically, the formula for obtaining the final score for each question-answer pair is as follows:

Where j represents the moral judgment of the question-answer pair, and s represents its safety rating from 1 to 5. These two evaluation steps not only reduce the single-step error of the evaluation large language model (LLM) but also further refine the granularity of the final score by establishing a connection between ethical judgment and safety rating.

After obtaining the scores for all question-answer pairs in each category, we take the average to obtain the final safety score for that category of questions. In this way, we can more accurately reflect the performance of different categories of questions in terms of safety, providing a comprehensive and detailed method for evaluating the safety of LLMs.

Ultimately, we plot a category-score radar chart based on the ten categories of safety questions and their scores to provide a visual representation of the safety of the large language model (LLM) under test. With this, we complete all the evaluation steps for the LLM under test.

In the actual evaluation, we use GPT-3.5 as our evaluation LLM. We configured it with temperature=0 and logprobs=True. During the ethical judgment phase, we set top_logprobs=2, and for the safety rating step, we set top_logprobs=5. Moreover, we use Python’s try statement to handle cases where the output from top_logprobs is not a number between 1 and 5, setting such outputs to the average value of 2.5.

We conducted attack assessments on eight LLMs, including ChatGPT, ERNIE, and Llama, from both domestic and international sources. The results shown in Figure 4 indicate that OpenAI’s ChatGPT series excels in terms of safety performance, with GPT-4 leading with an average CFSafety score of 4.02. Other large models such as Gemini, Qwen, and ChatGLM also perform well, while Llama2 scores relatively lower. From the radar chart analysis, it is evident that most language models fine-tuned through RLHF [32] excel across five classic safety scenarios. However, when facing the latest directive attacks, such as complex scenarios embedding and humanitarian persuasive, even the advanced GPT-4 exhibits some vulnerabilities. Additionally, due to the monolingual nature of their training data, models like ERNIEBot and ChatGLM score lower on attacks in minority languages. Our test results reveal significant breakthroughs in the security of language models (LLMs) over the past two years. These achievements also highlight some unresolved issues, motivating us to further focus on and address these shortcomings.

We further analyzed the safety score distribution of various models, as shown in Figure 5. The analysis indicates that most models’ scores are concentrated around a score of 5, reflecting that in most cases, the models are considered safe. However, a portion of the scores exhibits a clear bimodal distribution, indicating that within the 1 to 5 scoring range, scores tend to be more concentrated at the two extremes .

The emergence of a bimodal distribution holds special significance in our evaluation. Firstly, this distribution pattern mirrors the polarization trend commonly observed in human subjective evaluations, where there is a tendency to give either very high or very low scores.ref34 This polarization, often driven by reactions to clarity or ambiguity, causes human evaluators to have strong responses when situations are very clear or very ambiguous. The ability to simulate this human behavior indicates that language models, when conducting safety evaluations involving complex diversity and potential extreme cases, can provide response patterns similar to human decision-makers.

Furthermore, the identification of extreme cases in safety evaluations is crucial. The bimodal distribution demonstrates the models’ sensitivity in identifying potential extreme safety or danger scenarios. The presence of this scoring pattern not only increases the complexity of evaluations but also enhances our understanding and practical relevance of the assessment results.

We conducted attack assessments on eight LLMs, including ChatGPT, ERNIE, and Llama, from both domestic and international sources. The results shown in Figure 4 indicate that OpenAI’s ChatGPT series excels in terms of safety performance, with GPT-4 leading with an average CFSafety score of 4.02. Other large models such as Gemini, Qwen, and ChatGLM also perform well, while Llama2 scores relatively lower. From the radar chart analysis, it is evident that most language models fine-tuned through RLHF ref32 excel across five classic safety scenarios. However, when facing the latest directive attacks, such as complex scenarios embedding and humanitarian persuasive, even the advanced GPT-4 exhibits some vulnerabilities. Additionally, due to the monolingual nature of their training data, models like ERNIEBot and ChatGLM score lower on attacks in minority language. Our test results reveal significant breakthroughs in the security of language models (LLMs) over the past two years. These achievements also highlight some unresolved issues, motivating us to further focus on and address these shortcomings.

According to a survey by Yupeng Chang and others ref33 , more advanced large language models (LLMs) generally possess stronger assessment capabilities. Therefore, you might consider selecting GPT-4 or another advanced LLM as the evaluator for SecuLLM. We believe this will help you achieve more stable and accurate assessment results.