RITFIS: Robust input testing framework for LLMs-based intelligent software

Intelligent software based on large language models (LLMs), such as ChatGPT¹¹1https://openai.com/chatgpt, New Being²²2https://www.bing.com, Chatsonic³³3https://writesonic.com, and Paradot⁴⁴4https://www.paradot.ai, has garnered widespread attention due to its exceptional semantic understanding and text generation capabilities. Humans can collaborate with LLMs through ‘prompt + example’ as input [1], thereby accomplishing many security-relevant natural language processing (NLP) downstream tasks, such as financial sentiment analysis [2], public opinion monitoring [3], and fraud detection [4], among others. Taking financial sentiment analysis as an example, the financial market generates a massive amount of news, reports, and social media content, making manual analysis of these texts impractical. Investors and financial analysts need to understand market sentiment to predict the performance of stocks, bonds, or other financial products. Therefore, it is necessary to collect relevant financial textual data, apply models to classify the sentiments of untagged financial texts, and use this for analysis and prediction. If influenced by adversarial texts, leading to inaccurate text classification, this can result in incorrect market sentiment judgment, misguiding investors and decision-makers into making unfavorable investment decisions, potentially leading to financial losses, reputation damage, market turbulence, and other severe consequences. Additionally, in the United States alone, the labor force for software testing is estimated to cost $48 billion annually [5]. For the development of stable-performing LLM-based intelligent software, and to ensure the quality of software responses, automated robustness testing of such software is crucial [6].

Current research focuses on the high sensitivity of LLMs to prompts [7]. Zhu et al. [1] have assessed the robustness of LLMs to adversarial prompts by dynamically creating minor perturbations in characters, words, and sentences. Liu et al. [8] have evaluated the robustness of LLMs to typographical errors in prompts using the Justice dataset. We believe that studying the overall input robustness of LLMs (prompt + example) is more important, as real-world inputs are often complex and varied [9]. Evaluating the model’s robustness to overall input is essential to comprehensively measure its performance, including understanding complex contexts, handling various input errors, and adapting to non-standard usages.

Therefore, this paper proposes RITFIS, a robust input testing framework for LLM-based intelligent software. To our knowledge, RITFIS is the first framework designed to evaluate the robustness of LLM-based intelligent software to natural language inputs. Inspired by the work of Morris et al. [10], RITFIS consists of objective functions, perturbations, constraints, and search methods. It transfers 17 automated testing methods aimed at the deep neural network (DNN)-based intelligent software to the LLM-based intelligent software testing scenario, covering character-level, word-level, sentence-level, and combination-level perturbations. Considering the strong generalization capability of large models and the fact that many companies and individual users will not or cannot fine-tune large models for specific domains due to insufficient data [11], RITFIS focuses on zero-shot situations. We conducted robustness tests on the famous open-source large model Llama [12] using five of the latest or commonly used DNN-based intelligent software testing methods on three datasets that include different language styles and industry terminologies. Experimental results show that current testing methods can reveal certain robustness flaws in LLM-based intelligent software, but there are limitations in testing capabilities.

Figure 1 outlines the proposed RITFIS, which aims to accomplish such testing: given the required threat model and prompt for testing, different testing methods are executed to automatically generate test cases for the original examples in the dataset. The testing method can be defined as a combinatorial optimization problem: using a goal function to define successful test cases, establishing a perturbation space for the original examples based on the method of perturbation, and using search methods to find a test case that meets the test objectives and certain language constraints. The modular design of RITFIS allows us to implement many different testing methods within a shared framework. The following is a detailed introduction to the basic modules of RITFIS: goal function, perturbation, constraint, and search method.

To assess the testing performance of existing automated testing methods on LLM-based intelligent software, we implemented 17 robustness testing methods through RITFIS. We initially evaluated the performance and efficiency of all methods, then selected five commonly used or latest testing methods for further experiments and analysis: TextFooler [18], StressTest [19], Checklist [20], TextBugger [21], and PWWS [22]. Experiments were conducted on three datasets, covering three scenarios: financial sentiment analysis (Financial) [23], movie review analysis (MR)⁵⁵5https://huggingface.co/datasets/MrbBakh/Rotten_Tomatoes, and news classification (AG’s News) [24], encompassing binary and multiple classifications as well as different text lengths. We used Llama-2-70b⁶⁶6https://github.com/facebookresearch/llama as the threat model for the experiments. Composed of 70 billion parameters, Llama-2-70b demonstrates exceptional accuracy and adaptability when processing large volumes of complex data, making it an important milestone in the current field of NLP. The evaluation metrics of the experiment include:

1) Success rate (S-rate) [10], which represents the proportion of usable test cases generated by the test method among all tested examples. In this experiment, its formula can be expressed as follows:

where, $N_{suc}$ is the number of test cases that test threat models successfully, and $N$ is the total number of input examples ($N$ = 1,000 in our experiment) for the current test method.

2) Change rate (C-rate) [10], which represents the average proportion of the changed words in the original text. C-rate can be expressed as:

where $\operatorname{diff}T_{k}$ represents the number of words replaced in the input text $T_{k}$ and $\operatorname{len}$($*$) represents the sequence length. C-rate is an indicator designed to measure the difference in content between the generated test cases and the original examples.

3) Perplexity (PPL) [10], an indicator used to assess the fluency of textual test cases. Perplexity is defined as the exponentiated average negative log-likelihood of a sequence. If we have a tokenized sequence $X$=($x_{0}$,$x_{1}$,…,$x_{t}$), then the perplexity of $X$ is,

where $\log p_{\theta}\left(x_{i}\mid x_{<i}\right)$ is the log-likelihood of the $i$-th token conditioned on the preceding tokens $x_{<i}$ according to the language model. Intuitively, given the language model for computing PPL, the more fluent the test case, the less confusing it is.

4) Time overhead (T-O) [10], which refers to the average time it takes for a test method to generate a successful test case.

5) Query number (Q-N) [10], which indicates the average number of times a population-based method is needed to query the threat model when generating a test case. The query number and the time overhead together reflect the efficiency of the test method.

In this study, we chose the text classification task as the focus for testing the input robustness of LLMs. Text classification, as a widely applied downstream task in the NLP field, aims to assign textual content to predefined category labels, such as ‘positive’ or ‘negative’. Recently observed, with LLMs outperforming traditional DNNs [25, 26] and crowdsourced workers [27, 28] in text classification, an increasing number of enterprises are inclined to use LLM-based intelligent software for text classification tasks. Table I presents the test results of five robustness testing methods on three datasets for 1000 randomly extracted original examples, with the best value for each metric highlighted in bold.

Analyzing from the perspective of the effectiveness of the testing methods in uncovering robustness flaws, these methods face the same difficulty as when testing DNNs-based software: the longer the average length of the text in the dataset, the lower the success rate of testing. For example, TextFooler, a commonly used testing method, achieved a 63.856% success rate on the shortest MR dataset, but only 48.143% on the longest AG’s News dataset. The PWWS method averaged a 63.972% success rate across the three datasets, performing better than other baseline testing methods. Notably, PWWS achieved an 80.308% success rate on traditional DNNs, indicating that existing methods can reveal the robustness flaws of LLM-based software to a certain extent, but their testing capability for such software is still limited. We believe this issue primarily stems from two factors: First, the complexity and high integration of LLM-based software may make it difficult for standard robustness testing methods to fully reveal its potential flaws. Second, most existing testing methods rely on fixed perturbation paradigms, and the dynamic, continuous learning characteristics of LLM-based software may mean that test results at a given time point do not reflect its long-term robustness. To address these issues, we suggest improving the establishment of perturbation spaces and search methods in testing algorithms, tailored to LLMs’ unique characteristics and behavioral patterns, thereby increasing the coverage and depth of testing. Secondly, considering the dynamic nature of LLM software, we need to adopt continuous, iterative testing methods and adaptive testing strategies to capture the behavioral changes of the software in long-term operation.

To further analyze the performance of the testing methods, this paper evaluates the quality of test cases generated by different testing methods through two indicators: change rate and text perplexity. All five testing methods misled LLMs with no more than an 11% change rate, indicating that slight perturbations to the original samples could lead to erroneous outputs by LLMs. In terms of PPL, test cases generated by StressTest performed best, with an average PPL score of 45.830, meaning these test texts were more natural and fluent.

Besides the quality of test cases, the efficiency of testing methods is also a key focus of this study, including time cost and the number of queries. In these two metrics, StressTest performed best: it saved at least 103.402 seconds and reduced 27.707 queries to the threat model per successful test case compared to the next best method. However, this significant improvement in efficiency seems to come at the expense of the effectiveness of robustness testing, as StressTest had the lowest success rate across the three datasets. We believe this phenomenon is due to an overemphasis on rapid completion of testing, leading to a neglect of the comprehensiveness of test cases, thereby affecting the accuracy and reliability of test results. Secondly, excessively reducing threat model queries means insufficient exploration of software robustness flaws, lowering the possibility of uncovering complex issues. Therefore, when conducting robustness testing on LLM-based software, there is a need to rebalance the relationship between efficiency and effectiveness, ensuring that increasing the speed of testing does not sacrifice the depth and comprehensiveness of test cases. Specific actions can include increasing the depth of threat model queries to more comprehensively assess software robustness. Secondly, consider introducing more complex test case generation algorithms, which can enhance the coverage and quality of test cases while maintaining efficiency. We suggest adopting dynamic adjustment strategies, dynamically adjusting the allocation of time and query resources based on feedback during the testing process, thus optimizing the balance between efficiency and effectiveness.

In this paper, we present RITFIS, which evaluates the input robustness of LLM-based intelligent software in NLP. RITFIS formalizes robustness testing as a combinatorial optimization task, consisting of goal function, perturbation, constraint, and search method, and applies existing testing methods for DNN-based intelligent software to the robustness testing of LLM-based intelligent software. We selected five testing methods and conducted experiments on three datasets across five testing metrics. The research findings indicate that current LLM-based intelligent software is not robust enough, and the flaw detection methods of existing testing approaches are also not sufficiently effective. Therefore, in future work, we plan to adopt continuous iterative testing methods and rebalance efficiency and effectiveness, targeting the unique input paradigms and hallucination issues of LLMs, to design more effective robustness testing methods.