Prompt Sapper: A LLM-Empowered Production Tool for Building AI Chains

Building AI chains is a rapid prototyping process that creates custom AI services on top of foundation models. Foundation models make rapid prototyping of AI services feasible, as we no longer need to spend time and effort on data engineering and model training, but focus on the act of solving problems using AI. Furthermore, rapid prototyping allows for the quick delivery of working software and obtaining feedback to iteratively improve AI services. As AI chain engineering frees us from low-level coding, we will see a revival of important but undervalued software engineering activities, such as requirement analysis, specification and verification (Meyer, 2022) throughout AI chain engineering. The key is to align task needs and problem understanding with model capability through iterative design, evaluation and improvement of AI chains and prompts. This process can be enhanced by the interaction with large language models, which we call “magic enhancing magic” in Section 3.2.

An AI chain consist of a set of cooperating function units, which we call workers, analogous to objects. AI chain is a recursive concept. Depending on the task complexity and model capability, an AI chain may organize cooperating workers into a hierarchy of workers. The worker hierarchy can be represented in composite pattern (Gamma et al., 1995), in which the leaf worker has no children and implements the functionality directly, and the composite worker (analogous to module) maintains a container of child workers (leaf or composite) and forwards requests to these children. An AI chain also needs to specify the workflow, control structure, and cooperation of its workers. Workers need to define a “function signature” for communicating with human and other workers. We can use workflow patterns (Initiative, 2023) to define and implement cooperation between workers, data management, and exception handling. These worker design aspects are analogous to system design and should follow computational thinking principles, such as problem decomposition, pattern recognition, and algorithmic thinking, as well as well-recognized software engineering practices, such as separation of concerns and modularity.

A prompt defines the function of a worker and “program” the foundation model to complete the corresponding task. The workers should follow single responsibility principle and play distinct roles instructed by their prompts.

While prompt programming lowers the barrier for non-technical individuals to develop AI prototypes, the difficulty of designing effective prompts increases with increasing reasoning ability of the worker. Some challenges arise in finding or generating representative few-shot examples (so called example sourcing (Jiang et al., 2022)), requiring AI chain engineers to be creative and maintain example datasets (although would be smaller than those typically used to train neural networks). Another challenge is to accurately describe task workflows, which can be achieved using semi-structured or code-like prompts (Singh et al., 2022; Madaan et al., 2022; Mishra et al., 2021).

By decomposing complex workflows into cooperating workers, it significantly reduces prompt design difficulty as each worker only needs to execute a simple sub-step of the complex workflow. Furthermore, we can provide each sub-task worker with sufficient specific examples, test and debug individual workers (akin to unit testing) to optimize its performance, and seamlessly plug the improved worker into the AI chain, as a systematic way to improve performance on the complex task. A well-design worker can be reused across multiple tasks.

Like other software projects, AI chain engineers often start with a vague understanding of “what they want”. Building software on such vague statements can be a major reason for the failure of software projects. We need to interact with engineers through requirement elicitation and gradually clarify the initial “what they want” statements into specific AI chain needs. This challenging task can be supported by large language models. By providing some examples of requirement elicitation, such as good open-ended requirement elicitation questions in the Software Requirements textbook (Wiegers and Beatty, 2013) by Karl E. Wiegers, the LLMs would learn to ask good open-ended questions for specific tasks and thus elicit specific AI chain needs. The Design view of our Sapper IDE (see Section 4.1) is equipped with such a requirement elicitation co-pilot that interacts with the engineer to elicit and analyze tasks requirements.

Mechanical Sympathy means that a racing driver does not need to be a mechanical engineer, but understanding how the car works can make one a better racing driver. The same applies to developing AI chains and writing prompts, especially as large language models are transforming AI from traditional design and engineering to something more akin to natural science (Kambhampati, 2022). Therefore, we need to empirically explore and understand emergent AI behaviors and capabilities.

Gwern Branwen’s blog (OpenAI, 2023) proposes that we need to anthropomorphize prompts. We need to consider how people would typically express the information we need if they had already discussed it on the Internet. Preliminary research (Gonen et al., 2022) suggests that the more familiar the model is with the language used in the prompt, the better the prompt’s performance tends to be. We also need to test different prompts to see what different outputs they may lead to and reverse engineer how large language models understand these prompts, thereby discovering any discrepancies between what we assume/expect and what the models understand. Understanding these discrepancies can guide us in decomposing tasks and writing prompts to fit with the model capability.

Foundation models provide playgrounds for experimenting prompts. Research has shown that large language models can create human-level prompts (Zhou et al., 2023). Inspired by this, our IDE’s Design view (Section 4.1) implements a ChatGPT-based prompting co-pilot to create candidate prompts for the generated AI chain steps as initial task decomposition (Section 3.3.2).

The initial AI chain requirements provided by AI chain engineers are often incomplete or ambiguous. Building the AI chain on such requirements makes it difficult to “build the right AI chain”. In software projects, it is usually necessary to clarify and refine requirements through communication between product managers and users. We achieve this through an LLM-powered requirement co-pilot, which leverages its vast knowledge to iteratively rewrites and expands the AI chain requirements.

When given a complex multiplication, such as 67*56, many people may not be able to provide the answer immediately. However, this does not mean that humans lack the ability to perform two-digit multiplication. With a little time, pen, and paper, it is not too difficult to stepwise calculate 60*50+7*50+60*6+7*6 and arrive at the final answer. The same principle applies when working with foundation models. When a model cannot solve a problem successfully in a generative pass, we could consider how humans would divide and conquer such complex problems. Consulting large language models can often provide inspiration for task decomposition.

We can iteratively break down the problem into small, simpler ones. If a sub-task is still too complex, it can be further decomposed into simpler sub-tasks until each small problem can be handled by the model in a single generative pass. This results in a hierarchy of sub-tasks, each of which is accomplished by an AI chain worker. We achieve this through an LLM-powered decomposition co-pilot, which facilitates the process of breaking down complex tasks into manageable sub-tasks.

AI chain workers collaborate to complete tasks according to a workflow. This requires algorithmic thinking to develop a step-by-step process for solving problems.

The AI chain workflow needs to support control structures, such as condition and loop. For example, in order to avoid introducing errors into correct code, PCR-Chain (Huang et al., 2023a) only attempts to fix the code when the LLM determines that there are last-mile errors (judging whether the code has errors is easier than fixing the code). Re3 (Yang et al., 2022) works by iteratively expanding a story based on an outline, while maintaining coherence and consistency throughout the story. Meanwhile, each worker needs to define a function signature in terms of input and output as well as any pre/post-condition constraints, akin to interface specification in software engineering, so that workers can interact or communicate with one another.

Users can create a new AI chain project (Create Project), download the current project to local disk (Download Project), or open a project on local disk in the IDE (Open Project). Users can download the backend Python code implementing the AI chain (Download Code) to local disk and reuse the code in their other software projects. Note that our sapperchain Python library (which is not open sourced) is required to execute the downloaded AI chain code.

Similar to the value of code in traditional software, prompts are the core assets in the AI chain paradigm, which can be reused across AI chain projects. Therefore, our Sapper IDE supports a Prompt Hub (Fig. 2  ) that is independent of project management. Prompt Hub has two tabs: Prompt Builder and Prompt Base.

In the Prompt Builder tab, the user can manually create new prompts. They can also drag an existing prompt from the Prompt Base tab to the prompt editor to edit it. Editing prompt blocks in the prompt editor is the same as editing other types of blocks in the AI chain editor. Experienced users can directly enter or edit prompt text in the Prompt Text view at the bottom. Users can also create or edit prompts in a structured way by the four prompt aspects (Context, Instruction, Examples, and Output Formatter). Except for the Instruction, the other three aspects are optional, but are recommended to represent the prompt in a more structure form. Clicking ”Save to Prompt Base” will save the created prompts to the Prompt Base or update existing prompts in the Prompt Base. All prompts can be viewed in the Prompt Base tab.

In the Prompts tab of an AI chain project, the user clicks the “Import Prompt…” button, which will open the Prompt Hub view. The user can then drag the prompts that the project will use from the Prompt Base tab to the prompt editor. Then, clicking the “Export to Project” button will export the selected prompts to the project. The user can change prompts used in the project in the project’s Prompt Console tab, but these changes will not affect the original prompts in the Prompt Hub. The user can download the prompts into local files or upload local prompt files to the IDE.

Workers use different types of engines to complete tasks, such as foundation models and external APIs. These engines can also be used across AI chain projects, so the AI chain IDE supports separate Engine Management (Fig. 2  ). Our IDE currently pre-installs three foundation models, including gpt-3.5-turbo, text-davinci-003, and DALL-E, and the Python standard REPL shell.

In the Foundation Engine tab, users can create a foundation model engine and config it to use different model parameters, such as temperature, maximum length, Top P, frequency penalty, presence penalty. In this way, users can create different engine instances of the same foundation model. Clicking “Save Engine” in the toolbar will save the created engine in the FM Engine tab, which can be edited later or exported to the projects for using.

In the Engines tab of an AI chain project, the user clicks the “Import Engine…” button, which will open the Engine Management view. The user can then drag the engines that the project will use to the engine editor. Then, clicking the “Export to Project” button will export the selected engines to the project. The user can change the configuration of the engines used in the project in the project’s Engine tab, but these changes will not affect the original engines in the Engine Management. The user can download the engine information to local files or upload it from local files to the IDE.

We recruited 18 participants for the study, of which all aged 18-25 years. Seven participants major in Computer Science, three major in Software Engineering, and two major in Artificial Intelligence. Of the 18 participants, 10 participants were novice programmers (0-1 year of experience), six were beginners (1-3 years of experience), and two were experienced programmers (3+ years of experience). In terms of gender, there were 10 male and 8 female participants. None of them have used visual programming tools before, and all have learned Python and use PyCharm before.

We conducted a within-subject study, in which each participant was asked to use three tools, native Python (PyCharm), Sapper with Design View (Sapper V2), and Sapper without Design View (Sapper V1), to program tasks in counterbalanced order. Counterbalanced order was employed to evenly distribute the potential sequence effects or learning biases associated with using different programming tools across participants, thereby minimizing the influence of order-related biases (Cook et al., 2002). In other words, since we had three different tools to choose from, there were a total of six possible combinations of orders in which the tools could be used. Each combination of orders was used by three participants in the study.

Before starting the formal tasks, we provided a training session to introduce how to use Sapper with Design View. We also gave them some warm-up tasks to practice and ensure that they understand the basic usage. During the training session, participants were encouraged to ask any questions they had about Sapper. In addition, we also ask them to program the warm-up tasks using Python to be fair.

Next, each participant was asked to use all three tools to program tasks in counterbalance order. We prepare three sets of tasks (Task A/B/C) with similar difficulties and each subtask in each set will assess the same programming constructs (i.e., plain use of LLM, if-else, while loop, variables & use of a different LLM). For each task, we provided a written document with the overall goal and general approach. We specify the type of LLM to use, and provide test cases for them to verify their programs.

Taking task A as example, where participants were tasked with developing a service for a question-answering robot. They need to design a simple system where users can pose questions. The system, powered by the text-davinci-003 model, would then provide the answers (plain use of LLM). Task A2 added the capability to distinguish between code-related and general questions, using distinct prompts for each (if-else). In Task A3, the system could continue answering questions if the participants wished (while loop). Finally, in Task A4, a variable is used to maintain the conversation history. After the participants ends the conversation, the participants need to use GPT-3.5-turbo model to summary all questions asked based on the history (variables & use of a different LLM). Task B/C, similar in complexity to Task A, involved the same programming constructs for each subtask. Further details regarding all tasks can be found on our GitHub repository²²2https://github.com/YuCheng1106/PromptSapper.

Doing this can reduce the potential bias introduced by irrelevant factors, such as one may include a testing phase while another may not. Once participants finished all tasks using all tools, they were asked to fill in a questionnaire regarding user experience and the cognitive dimensions for interface evaluation (Blackwell and Green, 2000). During the programming process, the computer screen was recorded to assist following analysis³³3We obtained their approval for this.

We utilize paired t-tests to assess the statistical differences among three tools. Given that our evaluation involves multiple tools (n=3), we will implement the Benjamini-Hochberg method (Benjamini and Hochberg, 1995) to adjust p-values for multiple inferential statistical tests. This approach helps in controlling the false discovery rate, ensuring more reliable conclusions. All p-values reported in our findings will be those that have undergone this correction process.

Sapper is easier to understand and use: We first assessed the time spent and correctness of both tools by measuring the total time taken to complete Tasks 1-4 and the number of correctly completed tasks. We performed paired t-tests to determine if there were any significant differences in time spent and correctness between the two tools. The null hypothesis for both tests was that the two tools have similar distributions.

For correctness, there is no significant difference between Sapper V2 and Python, with a t-statistic of -0.53 and a p-value of 0.59 (¿0.05). For time spent, the result indicated a significant difference in time spent using the two tools, with a t-statistic of 4.54 and a p-value of 0.0004 (¡0.05). This strongly rejects the null hypothesis and shows that participants took significantly less time to complete the tasks using Sapper V2 (Mean=1,689.00s, Std=447.49s) than Python (Mean=2,366.00s, Std=536.70s). We carefully examined the videos and revealed two main observations. First, we found that Python is more error-prone than Sapper V2. For instance, when using Python, participants made simple grammar errors such as forgetting to convert input into an integer or initiate a variable, regardless of their programming expertise. In comparison, Sapper V2 can save the time spent correcting these errors. Second, participants struggled to locate the correct LLM API in Python and spent a considerable amount of time reading documentation to find a proper API call format when using Python. This issue was exacerbated when multiple APIs were used in different formats, leading to confusion and further time consumption. For example, while participants could rely on the same package “openai” to invoke GPT3.5 and GPT3 models, the parameter names for input prompt changed from “prompt” in GPT3 to “messages” in GPT3.5, causing confusion among participants. In contrast, Sapper V2 encapsulates these complex API calls within a simple and integrated interface, reducing the time needed to locate and understand the correct API format. These observations show that our Sapper can provide more support to users by saving their time in learning programming language, debugging the syntax error, and instead can focus more on the logic and the functionalities.

Usefulness Scores: To assess the usability of Python and Sapper V2, we analyzed the results in terms of 9 different cognitive dimensions (Blackwell and Green, 2000), as depicted in Figure 3. The bar chart shows that Sapper V2 outperforms Python in terms of usability for all metrics, with higher scores indicating better usability.

We conducted a paired t-test for each metric and found that, except for diffuseness and visibility, no significant differences were observed between Python and Sapper V2 (with $p>0.05$) in all other metrics. Diffuseness refers to the conciseness of the notation used and how much space it requires to describe a task. A higher diffuseness score implies that the tool is more concise and requires less space to describe a task. Our statistical analysis (with $p={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}0.0151}<0.05$) revealed that Sapper V2 is significantly more concise and easier to use for participants, thus saving time and effort. Visibility is a measure of how easily users can see and find the different parts of the notation while creating or changing it. It also includes the ability to compare or combine different parts simultaneously. A higher visibility score indicates that the tool is easier to modify and create new components. The result (with $p={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}0.0006}<0.05$) suggests that Sapper V2 is particularly convenient when making changes to the code due to its high visibility score. These findings are also consistent with our previous observations that Sapper V2 simplifies the API calls and provides a more user-friendly interface, reducing the cognitive load on participants. Overall, our study suggests that Sapper V2 is a more usable tool.

Similarly, we conducted paired t-tests to evaluate correctness, the time spent and usability scores between Sapper V1 and Sapper V2 to understand the pros and cons of design view.

In terms of correctness, no significant difference was found between Sapper V1 and Sapper V2. No significance was found between Sapper V1 (Mean=1,751.00s, Std=493.34s) and Sapper V2 (Mean=1,689.00s, Std=447.49s) in time spent ($p={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}0.62}>0.05$). We carefully examine the usage videos from the participants. While Sapper V2 can quickly generate some basic blocks for simple tasks, its current situation also brings some new issues to the participants. For example, it currently does not support copy and paste so that the participants have to create similar blocks repeatedly. In comparison, V1 supports the copy-paste function and the participants can duplicate existing blocks and only need to slightly revise some prompts or models. Secondly, the current design view does not support automated generation of if-else or while loops, the participants should instead create one and perform draggings to put the related blocks under the condition or loops, which is similar to the block view in V1. Third, our task decomposition copilot requires a significant time to get the response causing extra time consumption. In addition to these, we surprisingly find that while the original purpose of the design view is for generating the basic blocks by given the task description, and then the participants can edit them in Block View, many participants instead directly use the design view to make changes, such as adding control flow in design view. However, as there are no significant time consumption differences observed, the current design view status can be used as an alternative to Sapper V1, and the users can choose the ones they prefer or use them interactively, which brings more flexibility and freedom to end-users.

For the usability scores, as seen in Figure 3, Sapper V1 obtains higher mean scores in most metrics except for premature commitment, diffuseness and viscosity. However, no significant statistic differences were found between Sapper V1 and Sapper V2 in each dimension (with all $p>0.05$), which indicates that these two variances of Sapper have similar user experience. The reason may be that the current design view is also another no-code interface that allows participants to generate basic blocks and add if-loop, while loop, which requires no coding background.

In terms of general usage of Sapper, we found that our Sapper provides great flexibility to users. In Block View, some participants adopt the hierarchy workflow, while some use a vertical workflow with a variable to record the value and feed it as input to another worker. In Design View, some use the design view to directly generate basic code given the requirement and then modify it; while some directly build up the structure manually in the design view.

In the first part of the study, we focused on evaluating the effectiveness of the Requirement Elicitation Copilot. Each participant was presented with a abstract task requirement: “I want to develop a service to help users prepare for interviews.” Their task was to utilize our tool for requirement elicitation within 10 minutes. Following this, participants were asked to think another task by themselves and clarify the requirement with our copilot within another 10-minute period. To gather valuable insights into their experiences, we administered a questionnaire that assessed user experience and interface evaluation dimensions. Specifically, we asked the participants to rate usefulness, serendipity, diversity, ease-of-use and relevance by using 5-point Likert scale (1 being the lowest and 5 being the highest). Usefulness refers to how effectively our Copilot assists participants in clarifying their requirements. It assesses whether participants feel more capable of understanding their needs after using our Copilot. Serendipity evaluates the unexpected discoveries or insights participants may come across while using our copilot, going beyond their initial expectations. Diversity assesses the variety of insights and recommendations provided by the Copilot during requirement clarification. Ease-of-use examines the Copilot’s user-friendliness, ensuring an efficient user experience. Relevance emphasizes the alignment of the Copilot’s queries and suggestions with participants’ specific requirements.

In the second part of the study, we aim to evaluate the effectiveness of the AI Chain Skeleton Generation Co-pilot. Each participant was tasked with using the task descriptions they had previously conceptualized and obtained through requirement elicitation in the earlier study to execute AI chain skeleton generation copilot and run this AI chain skeleton. Subsequently, they were asked to fill out another questionnaire to review the overall structure, completeness of content, and ease of modification of the decomposed AI chain skeleton using 5-point Likert scale (1 being the lowest and 5 being the highest). The evaluation of the overall structure examines whether the task decomposition provided a clear and coherent framework for participants to understand the task effectively. Completeness assesses whether all the task details, including sub-tasks and steps, were adequately considered during decomposition to prevent any oversights. Ease-of-modification gauges participants’ perception of how easily they could make adjustments to the decomposed structure, such as adding, removing, or editing sub-tasks or steps, to accommodate changes or optimize the task structure.

The 12 participants perform a total of 134 rounds of effective clarification question-answer sessions across the two requirement elicitation tasks (one provided by us and one self-conceived by participants). Among the 24 participant-task sessions, 2 sessions involve 4 rounds of clarification question-answer, 10 sessions involve 5 rounds, 8 sessions involve 6 rounds, and 4 sessions involve 7 rounds. The rounds of requirement elicitation question-answer are reasonable, considering the relatively short experimental time for each task, as well as the complexity of the requirement elicitation task.

In the 12 self-conceived requirement elicitation tasks by participants, various domains were covered, including mental health, diet, health, finance, career planning, academics, and creativity. Specifically, these task categories encompass emotional and mental health (3 tasks), financial planning and career development tasks (3 tasks), cooking and nutrition (2 tasks), health and fitness (2 tasks), academic guidance (1 task) and creative writing (1 task). More details about the tasks and the participant requirement elicitation process can be found in our GitHub repository⁴⁴4https://github.com/YuCheng1106/PromptSapper.

Figure 4 presents the evaluation results of the five aspects’ ratings for the requirement elicitation results by the 12 participants. For all these five aspects, the mean ratings are all larger or equal to 3.5, and the majority of scores fall in the range of 3 to 5. This indicates that our requirement elicitation copilot is satisfactory, helpful, easy to use, and provide diverse, unexpected and relevant suggestions. In detail, among all 24 requirement elicitation sessions they evaluated, participants rated the usefulness and diversity of the requirement elicitation results with scores of 4 or 5 in over 70% of the sessions. Additionally, in over 80% of the ratings, participants rated the ease-of-use and relevance of the requirement elicitation results with scores of 4 or 5. Moreover, for 11 of the ratings (45.83% of the total), participants rated the serendipity of the requirement elicitation results at 4 or 5.

We carefully checked the 24 requirement elicitation processes and find that our Copilot not only helped in clarifying and deepening their thoughts but also inspired them. For instance, when the participants use our tool to elicit the requirement for the mock interview tool, it first confirmed the objective, offering enhanced detail such as “an interview preparation service that provides interview questions and offers tips.” Furthermore, to spark inspiration, the co-pilot posed questions about potential features, suggesting, “Let’s consider how your service might evaluate a user’s performance.” Importantly, it refrains from repetitively asking similar questions within a single session. This effectiveness is attributed to our design, where the conversation histories are transmitted to the LLM during the whole session. Moreover, our chat-style interface is well-suited for a conversational question-answer context. Consequently, our Requirement Elicitation Copilot receives high ratings in these five aspects.

As shown in Figure 5, among all 24 tasks, all participants assigned scores of 3 or higher for all three metrics, with most ratings falling between 4 and 5. This indicates that our Copilot effectively assists users in generating AI chain skeletons in a complete and correct manner. We carefully reviewed and executed the AI chain skeletons generated by the 12 participants. Out of these, 18 could be directly executed to obtain results, and for the remaining six, minor modifications to the generated prompts were sufficient to achieve the desired outcomes. Upon inquiry, most participants mentioned that while the generated skeletons were executable and provided results, some adjustments were necessary. These adjustments might involve adding additional steps to refine the functionality or fine-tuning the prompts to align the service output more closely with their expectations. Nonetheless, they all expressed that the generated AI skeletons greatly facilitated their subsequent development efforts.