RestGPT: Connecting Large Language Models
with Real-World RESTful APIs

The emergence of recent powerful LLMs has enabled artificial intelligence systems to match human skills in utilizing tools [8, 9]. To enhance the performance of LLMs in accessing up-to-date information and carrying out precise mathematical reasoning, early work leverages simple tools like web search engines and calculators, such as ReAct [16], Toolformer [11], and ART [19]. Another line of research has focused on equipping LLMs to coordinate with external models for complex AI tasks, exemplified by HuggingGPT [13], ViperGPT [20], Visual ChatGPT [14] and Chameleon [12]. Recently, some work study how to enable open-sourced LLMs, such as LLaMa, to perform API usage [21, 15, 22]. Additionally, API-Bank [23] provides a systematic benchmark to showcase the efficacy of LLMs using tools to respond to human instructions.

Despite the notable advancements in incorporating tools for large language models, previous methods have exhibited certain limitations, most notably their restricted support for a limited number of specially designed APIs [12] and their inferior planning methods [9, 24, 12]. We compare RestGPT with other tool-augmented language models in Table 1. As shown, our work stands out by supporting for over 100 RESTful APIs. Furthermore, compared with most previous approaches adopt static offline planning which cannot interact with APIs and utilize feedback to adjust the plan, we employ a coarse-to-fine online planning framework with feedback, facilitating more flexible planning for complex instructions. Our work shares the similar spirit of AutoGPT, an autonomous agent capable of accomplishing complex tasks with numerous tools. While AutoGPT relies on developers to ensure compatibility with various applications, RestGPT can be integrated with any RESTful API-based applications in a plug-and-play fashion.

RESTful APIs have become a popular way to expose functionalities and data of web services to client applications [25, 17]. RESTful APIs also provide a standard for integrating external systems together with using a simple yet powerful interface. There are millions of RESTful APIs available on Internet, such as Spotify, Twitter, Gmail, etc. RESTful APIs are based on the REST architectural style, which emphasizes a client-server communication via stateless HTTP requests, including GET, POST, etc, where resources are identified by self-descriptive URIs [25]. The response of RESTful APIs are always structured in JSON format and contain various information. Thus, LLMs connected with RESTful APIs must possess a strong ability to extract the required information from the response.

OpenAPI Specification (OAS, or Swagger) [18], has been widely adopted as a standard for defining RESTful APIs. OAS is a structured documentation file which describes the endpoints, operations, parameters, response schemas, and other details of an API endpoint, providing a clear interface for our method to use the APIs.

As demonstrated in Figure 1, RestGPT is composed of three main modules: a Planner $\mathcal{P}$, an API Selector $\mathcal{S}$ and an Executor $\mathcal{E}$. The planner decomposes each user instruction into several sub-tasks, while the API selector selects APIs to address each sub-task. The executor, consisting of a Caller and a response Parser, performs RESTful API calls and extracts useful information from the JSON response to form the execution result. The core of each component is an LLM with the corresponding prompt and in-context examples describing the function of the component.

One of the challenges in connecting LLMs with a vast number of APIs is to ensure that the framework is able to fully understand the API documents with a limited context window size of LLMs. As depicted in Figure 1, we designate different modules to read distinct parts of the OpenAPI Specification (OAS). This strategy allows us to leverage OAS information to its fullest potentials when working with RESTful APIs. Specifically, the API selector reads the endpoint descriptions of all APIs to select a proper API for solving the current sub-task. Then, the caller uses the detailed documents of the API within the API plan to generate the correct API calling parameters and request body. Lastly, the parser is developed to make use of the response schema within OAS to generate the parsing code for information extraction.

To fully exploit the planning and decision making capabilities of LLMs and enable our method to dynamically adjust the plan to changing circumstances when accomplishing real-world user instructions, we propose a coarse-to-fine online planning mechanism in RestGPT.

The workflow of RestGPT can be characterized as an iterative “plan and execution” loop. During the planning stage, the planner and API selector collaborate to accomplish an instruction through iteratively decomposing it into suitable natural language sub-tasks and corresponding APIs. In each step $t$, the planner $\mathcal{P}$ leverages commonsense knowledge to generate a natural language (NL) sub-task $p_{t}$ based on the user instruction $q$, previous NL plans $(p_{1},...,p_{t-1})$, and execution results $(r_{1},...,r_{t-1})$, thereby constructing a high-level NL plan. Then, the API selector $\mathcal{S}$ reads the descriptions of available API endpoints to select appropriate APIs and construct the finer API plan $a_{t}$, which may contain a single or multiple API calls to solve the current NL plan $p_{t}$. Then the executor $\mathcal{E}$ executes the API plan $a_{t}$ and gets the execution result $r_{t}$ for current step. This process can be formulated as:

	NL Plan:	$\displaystyle p_{t}\leftarrow\mathcal{P}(q;p_{1},r_{1}...,p_{t-1},r_{t-1}),$		(1)
	API Plan:	$\displaystyle a_{t}\leftarrow\mathcal{S}(p_{t};r_{1},...,r_{t-1}),$
	Exec. Res.:	$\displaystyle r_{t}\leftarrow\mathcal{E}(a_{t};r_{1},...,r_{t-1}).$

In this way, the planner and API selector are dedicated to NL sub-task planning and API selection, respectively, effectively utilizing the large language model’s abilities of planning and text comprehension.

Alongside the “plan and execution” loop, we design two special states, “continual” and “end”, for the planner to monitor the execution result from the executor. Specifically, if the planner finds that the current executor’s output $r_{t}$ has not completed the present NL sub-task $p_{t}$, it will output a “continue” signal and provide a special NL plan $p_{t+1}$ to the API selector, instructing it to continue fulfilling the plan $p_{t}$. In such cases, the API selector will re-generate a new API plan based on the original NL plan $p_{t}$, new NL plan $p_{t+1}$, previous API plan $a_{t}$ and execution result $r_{t}$. This process is described as:

	API Plan:	$\displaystyle a_{t+1}\leftarrow\mathcal{S}(p_{t},p_{t+1};r_{1},...,r_{t-1};a_{t},r_{t}),$		(2)
	Exec. Res.:	$\displaystyle r_{t+1}\leftarrow\mathcal{E}(a_{t+1};r_{1},...,r_{t-1},r_{t}).$

If the planner assesses that the user’s request has been completed, it will give the termination signal “end” and output the final result. With such a design, our method achieves a more flexible online planning which is capable of handling various situations encountered in real-world scenarios.

The planner, API selector, and executor collaborate to form RestGPT’s coarse-to-fine online planning framework. This framework significantly enhances the ability to decompose tasks and select appropriate APIs, providing the model with the flexibility to effectively tackle user instructions.

Once an API calling plan is generated, the next step is to execute it. The executor $\mathcal{E}$ consists of a caller and a response parser. The caller should read the API documents carefully and generate correct parameters or request body for the API call. Due to the constraints of maximum context length, we filter API documents and only preserve APIs appearing in current API plan $a_{t}$. Given the generated parameters and request body, we use Requests Python library to call the RESTful API. Besides, to guide the response parser to extract information from the API response, the caller also generates a response description and output instruction for the response parser. Figure 2 presents an example output of the caller.

RESTful APIs typically return a JSON formatted response with much redundant information. The executor needs to extract the required information from the response and return it to the planner. However, the response may sometimes have a complex structure or be lengthy, making it difficult to extract important information via directly prompting the LLMs. To address this problem, we make use of the response schema defined in the OAS. Specifically, we utilize the coding capability of LLM to generate Python parsing code based on the provided schema and output instructions generated by the caller. Next, the Python code is executed to get the final result. If there are no execution exceptions or errors, the output is returned. Otherwise, the LLM is prompted to parse the response directly as a backup.

We select two common real-world scenarios: TMDB movie database and Spotify music player. The main consideration is to evaluate the capabilities of RestGPT: (1) augmenting LLMs with external specialized domain database via RESTful APIs; (2) connecting LLMs with RESTful APIs to autonomously control real-world applications. TMDB offers official RESTful APIs encompassing the information of movies, TVs, actors, and images. Spotify music player provides API endpoints to retrieve content metadata, receive recommendations, create and manage playlists, and control playback. For these two scenarios, we filter out 54 and 40 commonly used APIs respectively and obtain the corresponding OpenAPI Specifications to build RestBench.

High-quality instructions generally satisfy two crucial aspects: (1) to reflect a wide range of real user needs; (2) to cover different levels of complexity to fully study the reasoning and planning ability of our method. To achieve these goals, we adopt a bottom-up instruction collection approach. We employ 6 experts that work on NLP research to brainstorm instructions for different combinations of APIs. Along with the instructions, the experts need to annotate the gold API solution path for each instruction. To guarantee the quality of the instructions, we employ two additional experts to thoroughly verify the solvability of each instruction and correctness of the corresponding solution path. Ultimately, we annotate 10 instruction-solution pairs for each scenario as the development set, and 100 pairs for TMDB and 57 pairs for Spotify as the test set. Though the data scale is not large, these instructions are typical of the frequently raised user requests. Moreover, different from prior work which uses LLMs to get API calling procedure, we utilize human labeled API solution paths for evaluation. Table 3 presents example instructions of the two scenarios. The statistics of RestBench are shown in Table 2.

Since some user requests are time-dependent (see the TMDB example in Table 3), it is impractical to annotate a fixed ground-truth answer for each instruction, whereas, the API solution paths for most instructions remain consistent. If the model-generated API call path contains the gold API call path as a subsequence (with the elements not necessarily being contiguous), we think that the model has generated a correct path. To further evaluate the model’s performance, we rely on human evaluation to determine if the model result successfully fulfills the user query. We calculate the proportion of correct paths and successful query completions as metrics, i.e., Correct Path Rate and Success Rate. Moreover, the number of actual API calls can be utilized to measure the planning efficiency of different methods. Given the length of gold solutions, we further define $\Delta$ Solution Len. as the mean number of additional API calls required to successfully execute an instruction:

$$\Delta\text{Solution Len.}=\frac{1}{N_{s}}\sum_{i=0}^{N}(L^{i}_{real}-L^{i}_{gold})\cdot\mathbb{I}(i,\text{success}),$$

where $N_{s}$ is the number of successfully accomplished instructions, $L^{i}_{real}$ and $L^{i}_{gold}$ are the actually and gold number of API calls for the $i$-th instruction respectively, $\mathbb{I}(i,\text{success})$ denotes whether the $i$-th instruction is successfully completed.

We compare RestGPT with four recent baselines, including offline introspective method [9] used in HuggingGPT [13] and Chameleon [12], DEPS [7], ReAct [16] and Reflexion [26]. Since some methods are not originally designed for tool/API usage, we reproduce them and add the API executor proposed in Section 3.3 to make them able to call RESTful APIs. The maximum steps for DEPS is set to 10 and the maximum trials for Reflexion is set to 2.

To showcase the planning and API calling capabilities of our method, we implement two ablation variants of RestGPT. The first variant involves removing the planner and allowing the API selector to directly choose APIs in a ReAct style. This approach can be seen as ReAct equipped with our proposed executor. The second one is to replace the schema-based response parser with an LLM that directly reads and extracts the required information from the JSON response.

In our experiments, we employ text-davinci-003 from OpenAI as the LLM for RestGPT and all baselines. The decoding temperature is set to 0 for the most deterministic generation.

Table 4 shows the performance of RestGPT and baselines on two scenarios. Our approach outperforms all other methods in both scenarios, achieving a success rate of 75% on the movie database and over 70% on the music player. Note that in most cases, the correct path rate is slightly higher than success rate, indicating that the method may generate correct API calling plan but fail to execute it. RestGPT also stands out with its minimal solution length, showcasing the superior planning ability of the coarse-to-fine online planning mechanism.

Ablation experiments on coarse-to-fine planning and schema-based parser show both mechanisms are conductive to the model performance. Particularly, when removing the planner, the performance degrades significantly, indicating that current LLMs are unable to simultaneously conduct planning, API understanding and selection. Thus, the coarse-to-fine planning mechanism plays a crucial role in our framework. The ablation results without parser demonstrates that the schema-based parser enables LLMs to better comprehend and parse the real-world API responses with complicated structure.

To investigate the performance of our method with different base LLMs, we implement RestGPT with ChatGPT (gpt-3.5-turbo-0301), Llama2-13B (Llama-2-13b-chat-hf), and Vicuna-13B (vicuna-13b-v1.5). As shown in Table 4, the performance of ChatGPT is slightly worse than text-davinci-003. Interestingly, we have tried all official checkpoints of Llama2-13B, but none of them were able to comprehend the prompt and generate valid plans. In contrast, Vicuna-13B, which is fine-tuned from Llama2 on user-shared conversations, can accomplish some simple instructions. This result indicates that by fine-tuning LLMs on ChatGPT-generated data, the model can acquire the ability to understand and follow complicate prompts.

To further investigate the effectiveness of different modules in RestGPT, we conduct error analysis. In Figure 3, we classify errors based on the module in which they occur. We discover that the majority of errors occur during the planning stage, i.e., within the planner (purple) and API selector (blue). The planner sometimes loses track of its intended objective after multiple rounds of execution, resulting in early exit before completing the instruction. For the API selector, it may either select incorrect APIs or hallucinate to make up in-path parameters. This error analysis highlights the insufficient planning and decision-making capabilities of LLMs.

Compared with text-davinci-003, ChatGPT tends to make more errors in the planning stage, leading to slightly worse performance on both scenarios. More specifically, we find that ChatGPT is often too verbose and tend to continue planning even after the user instruction has been fulfilled. This behavior can be attributed to the fact that ChatGPT is trained specifically for conversational interactions, which encourages it to generate more lengthy responses.

In this section, we aim to demonstrate the scaling ability of RestGPT on two dimensions: scaling the difficulty of the tasks and scaling the number of APIs.

For each instruction in RestBench, the length of gold solution path indicates the complexity of the instruction. We calculate the success rate of models on instructions with varying complexities. As depicted in Figure 4 (a) (b), the success rate of all methods decreases as the complexity of the instruction increases. Notably, when the gold path length is 4, all baselines struggle to complete the task in both scenarios. In contrast, our proposed RestGPT can still achieve a success rate of over 40%, showing its superior performance in planning and API calling.

Before conducting experiments on scaling the number of APIs, we handpicked 10 APIs from TMDB and created a small test set comprising 15 instructions. All 15 instructions can be resolved using the selected 10 APIs. Then, we increasingly expanded the number of APIs and introduced additional noise APIs sourced from the official TMDB APIs. The results are shown in Figure 4 (c). As the number of noise APIs increases, the performance of all baseline methods deteriorates due to their inferior planning and reasoning. However, our method almost remains unaffected. These results effectively demonstrate the strong extensibility of our proposed RestGPT.

In Figure 5, we conduct a case study to compare the planning ability of RestGPT with the offline planning [9, 12] and ReAct [16] framework. Firstly, we observe the offline method is unable to solve most user instructions. As depicted in Figure 5 (a), the planner not only selects the wrong API (step 2), but also ignores the dependencies between APIs and used the parameter “user_id” before obtaining it (step 4). Regarding ReAct which generates chain-of-thought and actions in an interleaved manner, we find that current LLMs have a limited ability to simultaneously conduct planning, API understanding and selection. As shown in Figure 5 (b), the planner of ReAct generates a sub-task that is difficult to solve (step 2) and also ignores the dependencies between different APIs (step 3). Due to the inferior planning, it consumes 6 API calls to complete the task. In contrast, RestGPT employs a planner to generate high-level NL sub-tasks and an API selector to choose appropriate APIs to solve the sub-task. Notably, in step 3, the planner assesses the playlist that has not been successfully created and generate "continue" signal with further instructions for the API selector. Our method accomplishes the instruction with only 4 API calls. The coarse-to-fine online planning framework of RestGPT fully exploits the LLMs’ planning and document understanding capabilities, providing the model with the flexibility to tackle complex user requests.