The Potential of LLMs in Automating Software Testing: From Generation to Reporting

Agent-Based Software Testing (ABST) refers to the application of agent-based systems—encompassing software agents, intelligent agents, autonomous agents, and multi-agent systems—to address challenges in software testing. ABST aims to enhance the testing process by automating complex and resource-intensive testing tasks, thereby improving efficiency and accuracy [1]. This approach has attracted interest since 1999, with research activity reaching notable peaks over the past decade. The majority of ABST studies emphasize system-level testing, with Java serving as the predominant target language. Nevertheless, other programming languages, such as C, C++, Python, and Perl, have also been explored. In [2], the authors propose an agent-based framework designed for the automated testing of web-based systems. The framework incorporates multiple agents, including the Test Runtime Environment (TRE) agent, which serves as the central component of the system, as well as the Test Script Generator, Test Executor (TE) agents, and the Dashboard agent. These agents work collaboratively to execute and monitor the testing processes. Similarly, [11] introduces a multi-agent-based software environment tailored for web-based applications, in which agents can dynamically join or leave the system. Communication among agents is structured across three layers: the lowest layer facilitates message transmission between agents, the middle layer defines message content and ontology, and the highest layer employs communication protocols grounded in speech-act theory. An alternative approach is presented in [3], where the authors developed multiple agents to test an industrial coffee machine. Each agent represents a software module and employs fuzzy logic to prioritize software tests based on fault probabilities.

A study by [4] explores the practical application of Large Language Models in software testing within the software industry. Drawing insights from 83 completed questionnaires submitted by experienced and diverse professionals in the field, the research seeks to understand the extent and manner in which software testing practitioners utilize LLMs across different phases of the software testing lifecycle. The anonymous survey gathered data on participants’ usage of LLMs in testing activities, the specific tools employed, and their perspectives on the potential of LLMs in this domain. The findings indicate that while 52% of respondents have not yet utilized LLMs for testing, 48% have integrated these models into various activities, including requirements analysis, test plan development, and test automation. Tools such as ChatGPT and GitHub Copilot were frequently mentioned. The study concludes that LLMs hold significant promise as tools for enhancing software testing practices. However, the authors caution against indiscriminate usage, particularly in scenarios involving sensitive information, underscoring the need for careful consideration in such contexts. A recent comprehensive review by [12] analyzed 102 research papers, including 82 published in 2023, focused on the application of Large Language Models (LLMs) in software testing. The review highlights the utility of LLMs in mid-phase software testing activities, such as generating unit test cases, crafting test assertions, and producing system test inputs. Furthermore, LLMs are identified as valuable tools in later stages of the software testing lifecycle, providing support for bug analysis, debugging, and automated repair processes. In [5], the researchers introduced Kashef, a tool designed to assist software testers in designing, generating, and executing test cases, with a particular focus on web microservices applications. Kashef employs a multi-agent architecture comprising two agents powered by Large Language Models (LLMs)—the Test Engineer and the HTML Interpreter—alongside a non-LLM agent, the Code Executor. The tool integrates various LLMs, including GPT-3.5, GPT-4, CodeLlama, and Llama2, in conjunction with supporting libraries such as LangGraph for machine learning functionalities and Selenium for testing purposes.

As depicted in Fig. 1, the proposed architecture consists of four primary components, each playing a unique role in coordinating and enhancing the automated software testing framework.

1. 1.

   Audio WebClient: A foundational web application designed to capture user inputs, either through voice commands or text. Its primary role is to initiate the testing workflow by sending an HTTP GET request to the Software Testing Agent.

2. 2.

   Software Testing Agent: It serves as the system’s core component, hosting the primary logic and coordinating interactions between various components. The Software Testing Agent functions as an abstraction layer for smaller subcomponents tasked with key operations such as generating test scripts, executing tests, and creating bug reports.

3. 3.

   Large Language Models (LLMs): LLMs are integral to the proposed automated testing framework. The Software Testing Agent leverages LLMs for various tasks, including extracting key entities from user commands to streamline operations, generating customized unit tests tailored to specific requirements, and producing DOT graphs to visualize the application’s call graph.

4. 4.

   The Development Environment: It serves as the workspace housing the project code to be tested. This component facilitates automated testing by providing access to the target project, executing generated test cases, and displaying testing results and coverage metrics.

The low-level architecture, illustrated in Fig. 2, provides a detailed overview of the entire workflow, highlighting the responsibilities of each module and their interactions within the system. The process is initiated by the Client, typically through a voice command. The user prompt is received by the Test Generator API, which represents the entry point for the testing framework. The prompt is forwarded to the integrated LLMs, where entities such as the project name, the specific subfolder and the programming language of the project are extracted. Once the LLM identifies those entities, the FileLocator module is activated, which locates the specified project folder and locates the files within it.

Following that, the contents of these files are extracted and are sent as a prompt to the LLM, with the objective of generating unit tests and their accompanying rationale, using Gemini in our case. Additionally, the LLM generates a DOT graph string to help visualize a call graph of the code base interactions. After receiving the generated test scripts, the test files are created within the project. Next, the PDF Report Generator is activated, which as a first step orders the Test Executor to execute the newly created test files. Details results of the test runs, including code coverage metrics are provided by the Test Executor. Upon receiving this information, the Report Generator creates a comprehensive PDF report including test results, the rationales, code coverage and the generated call graph. This report is then returned to the user, finalizing the whole workflow.

The generated output of the system is a structured PDF report, presenting detailed insights regarding the testing process and results. It begins with listing the Test Rationale of each file and its functions. For every function, it describes the basic test cases and any applicable edge cases. This is followed by the Test Results section, which lists all executed test cases. Each test case is marked as Passed (highlighted in green) or Failed (highlighted in red), with additional details provided for any failures. Next, the Coverage Table summarizes the testing coverage, shoeing percentages, statements tested, and missed statements for each test file, along with an overall summary. Finally, the report concludes with a Call Graph that visualizes the code interactions. For future improvements, the call graph could be enhanced with a heatmap overlay, highlighting areas more prone to errors and requiring additional testing.

For example, in the cinema management project, the rationale for the ”test rent movie” function in the Library class includes the following scenarios:

• •

  Basic case: “Tests renting an available movie to an existing member.”

• •

  Edge cases: “Tests renting a non-existent movie, renting a movie to a non-existent member, and renting an already rented movie.”

In contrast, a simpler example can be seen in the library management project for the getTitle function in the Book class:

• •

  Basic case: “Tests retrieving the book title after object creation.”

• •

  Edge cases: Not applicable.

The system’s performance was assessed by logging detailed timing information for each execution, including the duration of operations such as component retrieval, unit test generation, and test execution. Additionally, test coverage and the overall execution status were recorded and categorized as either successful—when all steps, including test generation, execution, and PDF report generation, were completed successfully—or failed. Out of 20 executions on the Python projects, the system completed all runs without any failures. In contrast, 3 failures were observed out of 24 executions for the Java projects. These failures were primarily caused by ambiguous prompts, which hindered the accurate identification of necessary components, or by generated test scripts containing compilation errors, leading to failures during test execution.

The average total execution time for each run was approximately 83.5 seconds. The largest portion of this time, about 62.8 seconds on average, was dedicated to test generation by the LLM, which included generating rationales. This was followed by folder location, taking an average of 9.7 seconds, DOT graph generation at 5.4 seconds, and test execution at 3.2 seconds. Retrieving components required 1.3 seconds on average, while both PDF report creation and writing test files took less than a second each.

When comparing the average total execution time by programming language, Java executions averaged 86.7 seconds, while Python executions averaged 80 seconds. Interestingly, the time taken for test generation by the LLM was nearly identical for both languages, with Java averaging 62.4 seconds and Python 63.3 seconds. Most operations across the two programming languages showed similar average durations. However, a notable difference was observed in the test execution phase, where Java required an average of 5.44 seconds, compared to only 0.87 seconds for Python. This discrepancy in test execution time accounts for the gap in the overall average execution times between the two languages, as other operation times remained consistent, as shown in Fig. 3.

As anticipated, the average time for test generation varied significantly across projects, with more complex projects requiring substantially more time. For example, LibrarySystem averaged 92.54 seconds, Cinema 92.43 seconds, StudentManager 39.79 seconds, and Experiment 34.17 seconds. This variation is due to the greater number of files and increased logical complexity in the more complex projects. A summary of this analysis is presented in TABLE I.

However, test execution times were relatively consistent across projects of the same programming language, even though varying in complexity. As noted earlier, Java projects had higher execution times, with LibrarySystem averaging 5.39 seconds and StudentManager 5.48 seconds. In contrast, Python projects showed much lower execution times, with Cinema averaging 0.79 seconds and Experiment 0.96 seconds.

The overall test coverage across all projects was notably high, reflecting the effectiveness of the generated test cases. When grouped by project, the coverage percentages revealed some variation. StudentManager achieved full coverage at 100%, while Experiment followed closely with 98.6%. LibrarySystem exhibited slightly lower coverage at 94.67%, and Cinema recorded the lowest coverage at 88.3%. This is summarized in TABLE I.

When analyzing the results by programming language, Java projects demonstrated slightly higher average coverage at 97.71% compared to Python projects, which averaged 93.45%. This difference highlights potential language-specific factors influencing coverage, such as testing frameworks or execution times. Despite this variation, both languages consistently delivered high coverage, ensuring reliable testing across all projects.

To evaluate the performance of an alternative LLM, such as ChatGPT, we tasked it with generating test cases for the LibrarySystem project. The total generation time was approximately 3 minutes, which is twice as long as the average time observed with Gemini. Despite the longer duration, the test coverage achieved was impressively high at 98%. Notably, the ChatGPT Web Application was used for this task instead of the API, and the additional rendering process may have contributed to the increased generation time.