\sys: A Prompt Programming Language for Harmonious Integration of Programs and Large Language Model Prompts

(1) Readability and Flexibility. Natural language provides a readable and understandable human-LLM interface, while programming languages enable flexible controls, clear structures, modularization, reusability, and external tools. \sys aims to utilize both strengths.

(2) Easy Parallelization. Parallelization is a critical performance optimization for LLM text generation (Yu et al., 2022; Kwon et al., 2023). \sys aims to provide transparent support for parallelization with almost no extra code introduced.

(3) Convenient Conversion. \sys aims to facilitate easy conversion between structured data (including codes) used in programming and the natural language used in LLMs.

As illustrated in the \sys program of Figure 2(a), the syntax and semantics of \sys are extended from Python with slight modifications to integrate natural language prompts and LLMs calls into the traditional programming constructs deeply.

functions are the fundamental building blocks of \sys, denoted by the @ppl decorator. Each \sys function has an implicit context to store prompt-related information. Within \sys functions, the expression statements are interpreted as “interactions” with the context, where the typical behavior is to append strings to the prompt.

Expression statements are standalone Python statements formed by an expression, for example, "str", while an assignment (e.g., a = "str") is not. Their evaluation result is handled differently according to the environment. For example, it causes no extra effect in standard Python while the result is displayed in the interactive environment in Jupyter. Similarly, \sys provides an environment where each \sys function contains a scratchpad of prompts (i.e. context), and the values of expression statements will be converted to prompts (via promptify) and appended to the prompt scratchpad. For example, a standalone string as line 7 or 8 in Figure 2(a) will be appended to the prompt scratchpad.

LLM generations (gen), when being called, implicitly use the accumulated prompt in the context of the \sys function. A special case is the standalone f-string, which is split into parts and they are captured as prompts in order, detailed in Appendix A.1. This provides a more readable way to combine prompts and LLM generations in one line. For example, line 10 of Figure 2(a) means start generation by first adding the prefix “The name of the author is ” to the prompt, which matches human intuition.

Context managers modify how prompt statements are accumulated to the prompt in the context. \sys provides two types of context managers:
(1) Role Changers (such as AIRole in Figure 2(a)) are used in chats to specify the owner of the message, and can only be used as the outmost one.
(2) Prompt Compositors (such as NumberedList in Figure 4) specify the delimiter, indention, indexing format, prolog, and epilog. These compositors allow for meticulously structured finer-grained prompts in hierarchical formatting through native Python syntax, greatly enhancing flexibility without losing readability.

Definition is a special base class for custom definitions. As shown in Figure 4, once declared, Definitions can be easily referred to in the f-string, or used to create an instance by completing its description. They are useful for representing concepts that occur frequently in the prompt and support varying the instantiation in different prompts. See more usage in Appendix D when building a long prompt.

runtime context. Similar to Python’s runtime context that contains local and global variables (locals() and globals()), the context of \sys functions contains local and global prompts that can be accessed via records() and convo(). The example in Figure 3 illustrates how the prompts in the context changed during the execution.

Context passing across functions. When a \sys function (caller) calls another \sys function (callee), users might expect different behaviors for initializing the callee’s context depending on specific use cases. \sys provides four ways as shown in Figure 5:
(1) new: Creates a clean context. This is the default and most common way.
(2) copy: Creates a copy of the caller’s context. This is similar to “call by value” where the modification to the copy will not influence the original context. This is useful for creating independent and asynchronize LLM calls as shown in Figure 2(a).
(3) same: Uses the same context as the caller. This is similar to “call by reference” where the modifications are directly applied to the original context.
(4) resume: Resumes callee’s context from the last run. This makes the function stateful and suitable for building interactive agents with history (see Figure 9(b) for an example).

When decomposing a long prompt into smaller modules, both new and same can be used as shown in Figure 11, but the default way (new) should be prioritized when possible.

Asynchronous semantics. To automatically leverage potential independence among LLM calls and achieve efficient parallelization, we borrow the idea of the asynchronous semantics from PyTorch (Paszke et al., 2019). Asynchronous semantics means the system can initiate operations that run independently of the main execution thread, allowing for parallelism and often improving performance. In \sys, LLM calls (gen) are asynchronous, which means the main thread proceeds without being blocked by gen until accessing the generation results is necessary. See implementation details in Section 4.2.

Tool calling. Tool calling is an indispensable ability for LLM agents to interact with external resources. To streamline the process of providing tools to LLMs, \sys automatically creates tool specifications from Python functions by analyzing their docstrings and signatures (detailed in Appendix A.3). As shown in Figure 6, the Python functions is_lucky and search can be directly utilized as tools for LLMs. Therefore, thanks to the well-written docstrings, the functions from Python’s rich ecosystem can be integrated into LLMs as tools with minimal effort. Moreover, \sys automatically converts LLMs’ outputs as function calls that can be directly executed to obtain results.

Tracing and caching. \sys supports tracing \sys functions and LLM calls to facilitate users to understand and debug the program executions. The trace is useful for reproducing (potentially partial) execution results by loading cached responses of the LLM calls, which enables failure recovery and avoids the extra costs of resending these calls. This also unlocks the possibility of debugging one LLM call conveniently. Users can also visualize the program execution to analyze the logic, expense, and runtime of the program. \sys provides two modes of tracing, namely strict and non-strict modes, corresponding to different reproduction requirements (perfect or statistical reproduction). We further explain the implementation details for handling the asynchronous semantics in Appendix C.1, and demonstrate the trace visualization in Appendix C.2.

To provide the runtime context for \sys functions more smoothly and make it transparent to users, we choose to compile (more specifically, transpile) the Python source code to insert the context into the code. We use Python’s ast package to parse the original function as AST, mainly apply the following code transformations on the AST, and compile back from the modified AST to a Python function.

• •

  Context management. We use the keyword _ctx to represent the context of the function, which is inserted in both the definition of \sys functions and the function calls that need the context. The functions that require the context as inputs are annotated by the attribute __need_ctx__, including all functions introduced in Section 3.2: \sys functions, gen and context manager functions.

• •

  Capture expression statements. We add an execution function as a wrapper for all expression statements within the \sys function and let them “interact” with the context. The execution behavior differs based on the value types, where the behavior for the most common str type is appending it to the prompt of the context. For a standalone f-string, we split it into several statements to let them be “executed” in order so that gen functions in the f-string can use the text before them. This procedure is detailed in Appendix A.1.

To implement the asynchronous semantics, we adopt the concept of Future introduced in the Python standard library concurrent.futures, which represents the asynchronous execution of a callable. When the result of Future is accessed, it will wait for the asynchronous execution to finish and synchronize the result.

We introduce StringFuture, which represents a string object, but its content may not be ready yet. StringFutures behave almost identically to the normal str objects in Python, so that users can use them as normal strings transparently and enjoy the benefits of asynchronous execution. Meanwhile, the StringFuture delays its synchronization when possible, ideally only synchronizing when the str method is called.

Formally, we define StringFuture $S$ to be a list of StringFutures: $[s_{1},s_{2},\dots,s_{n}]$. Without loss of generality, we can regard a normal string as an already synchronized StringFuture. A StringFuture $S$ is synchronized when str($S$) is called, and it collapses to a normal string by waiting and concatenating the results of str($s_{i}$):

where $\oplus$ operation means (string) concatenation. Such representation enables delaying the concatenation ($\oplus$) of two StringFutures $S$ and $T$ as concatenating ($\oplus$) two lists:

For other methods that cannot be delayed, like len($S$), it falls back to materializing $S$ to a normal string and then calls the counterpart method.

Similarly, we introduce BooleanFuture to represent a boolean value that may not be synchronized yet, which can be used to represent the comparison result between two StringFutures. A BooleanFuture $B$ is synchronized only when bool(B) is called, which enables using future objects in control flows.

As shown in Figure 7, when implementing CoT-SC, \sys exhibits its conciseness and clarity compared with LMQL, SGLang, and Guidance. We mainly compare them from the following aspects:

Parallization. \sys performs automatic parallelization while being transparent to the programmer. In particular, since the returned value need not be immediately materialized, the different branches can execute concurrently. SGLang allows for parallelization between subqueries by explicitly calling the lm.fork() function. LMQL employs a conservative parallelization strategy, which waits for an async function immediately after it’s called. To support the same parallelization strategy, we use the asyncio library to manually launch LMQL queries in parallel at the top level. Guidance does not include asynchrony or parallelization as a core feature.

Generation and output retrieval. For LLM generation calls, \sys, SGLang, and Guidance use a Python function while LMQL uses inline variables (with annotations), which is less flexible for custom generation arguments. For the results of generations, both SGLang and Guidance store them as key-value pairs in a dictionary associated with the context object and are indexed with the assigned key when required. LMQL accesses the result by the declared inline variable, but the variable is not Python-native. In \sys, gen is an independent function call, whose return value can be assigned to a Python variable , which is Python-native and allows for better interoperability with IDEs. Moreover, \sys separates the prompt reading and writing for the generation. Users have the option to write generation results back to the prompt or not (called without side-effects in Table 1). This allows a simple replication of LLM generation calls sharing the same prompt. Otherwise, explicit copying or rebuilding of the prompt would be necessary for such requests.

Context management and prompt capturing. Both SGLang and Guidance need to explicitly manage context objects (s and lm) as highlighted as red in Figure 7. The prompts are captured manually using the += operation. The branching of the context is implemented differently, where SGLang uses the explicit fork method and Guidance returns a new context object in the overridden += operation. In contrast, \sys automatically captures standalone expression statements into the prompt context, similar to LMQL which captures standalone strings. The context management is also done automatically for nested function calls in \sys and LMQL, where \sys four patterns new, copy, same, and resume are supported in \sys (see Figures 5 and 2(a)) while LMQL only supports new and copy.

We validated the effectiveness of parallelizing LLM calls in \sys in three tasks: question answering with CoT-SC, content generation with skeleton-of-thought (SoT) (Ning et al., 2023), and hierarchical summarization with MemWalker (Chen et al., 2023). As shown in Table 2, the parallel version achieves significant speedup across different branching patterns, and the actual speedup ratios almost match the estimated ones. More experimental details can be found in Appendix B.

We use the ReAct (Yao et al., 2023) tool-use agent as an example to compare the accessibility of tool calls in different languages. As shown in Figure 8(a), the major workflow of ReAct unfolds over several cycles, each comprising three stages: (1) generating “thoughts” derived from existing information to steer subsequent actions; (2) selecting appropriate actions (tool calls) to execute; (3) retrieving the results of executing the actions. These results then serve as observations and influence the subsequent cycle.

LLM Tool calling can be divided into three steps: 1) encoding the external tools as specifications that are understandable by LLMs, 2) parsing generated tool calls, and 3) executing the tool calls. \sys provides out-of-the-box support for these steps based on the OpenAI’s API (OpenAI, ). As shown in Figure 8(a), documented Python functions like is_lucky defined in Figure 6 can directly be used as the tools argument without manual efforts. Differently, in Figure 8(b), Guidance (similarly for SGLang and LMQL) can generate tool calls and parse structured arguments via its select and other constraint primitives, but the tool specifications and format constraints need to be written manually. Guidance provides another way to automate this process but with limited support that only works for functions with string-typed arguments.

We consider a simplified yet illustrative scenario of agent communities (Park et al., 2023; Hong et al., 2023; Qian et al., 2023), where two agents chat with each other, as demonstrated in Figure 9(c). During the chat, each agent receives messages from the counterpart, maintains their own chat history, and makes responses. \sys provides flexible context-passing methods for implementing such agents and we illustrate two ways in Figure 9 as examples.

(a) Explicit chat history: As shown in Figure 9(a), the conversation history is explicitly stored in self._history. The history is first initialized in the _setup function. In the chat function, the history is first retrieved and written into a new prompt context. At the end, self._history is refreshed with records() that represents updated conversation with new messages and responses.

(b) resume the context: As shown in Figure 9(b), each call to the chat method will resume its memorized context, allowing continually appending the conversation to the context. The context is initialization in the first call by copying the caller’s context. This approach provides a clear, streamlined, concise way to manage its own context.

As demonstrated in Figure 7, \sys is more concise than other compared languages to implement the Cot-SC algorithm. Quantitatively, we further introduce a new metric AST-size to indicate the succinctness of programs written in different languages, which counts the number of nodes in the abstract syntax tree (AST) of the code. For Python programs, we can obtain their AST using the standard library with ast.parse.

As shown in Table 3, LMQL, SGLang, and Guidance need about twice AST nodes than \sys for implementing the same tasks, highlighting the succinctness of \sys.