Deploying Foundation Model Powered Agent Services: A Survey

Many research works focus on system optimization to deploy machine learning models in edge-cloud environments. KACHRIS reviews some hardware accelerators for Large Language Models (LLMs) to address the computational challenges [4]. Tang et al. summarize scheduling methods designed for optimizing both network and computing resources [5]. Miao et al. present some acceleration methods to improve the efficiency of LLMs [6]. This survey includes system optimizations, such as memory management and kernel optimization, as well as algorithm optimizations, such as architectural design and compression algorithms, to accelerate model inference. Xu et al. focus on the deployment of Artificial Intelligence-Generated Content (AIGC), and they provide an overview of mobile network optimization for AIGC, covering the processes of dataset collection, AIGC pre-training, AIGC fine-tuning, and AIGC inference [7]. Djigal et al. investigate the application of machine learning and deep learning techniques in resource allocation for Multi-access Edge Computing (MEC) systems [8]. The survey includes resource offloading, resource scheduling, and collaborative allocation. Many research works propose different algorithms to optimize the design of FMs and agents.  [1],  [9] and  [10] present popular FMs, especially LLMs.  [11],  [12] and  [13] summarizes model compression and inference acceleration methods for LLM.  [2],  [14], and  [15] review the challenges and progress for the development of agents.

In summary, the above studies either optimize edge-cloud resource allocation and scheduling for small models or design acceleration or efficiency methods for large FMs. To the best of our knowledge, this paper is the first comprehensive survey to review and discuss the deployment of real-time FM-powered agent services in heterogeneous devices, a research direction that has gained significant importance in recent years. We design a unified framework to fill this research gap and review current research works from different perspectives. This framework not only delineates essential techniques for the deployment of FMs but also identifies key components of FM-based agents and corresponding system optimizations specifically tailored for agent services.

This paper presents a comprehensive survey on the deployment of FM-powered agent services in edge-cloud environments, covering optimization approaches spanning from hardware to software layers. For the convenience of readers, we provide an outline of the survey in Figure 2. The contributions of this survey are summarized in the following aspects:

• •

  This survey proposes the first comprehensive framework to provide a deep understanding of the deployment of FM-powered agent services within the edge-cloud environment. Such a framework holds the potential to foster the advancement of AGI greatly.

• •

  From a low-level hardware perspective, we present research on various runtime optimization methods and resource allocation and scheduling methods. These techniques are designed to establish a reliable and flexible infrastructure for FMs.

• •

  From a high-level software perspective, we elucidate research efforts focused on model optimization and agent optimization, thereby offering diverse opportunities for building intelligent and lightweight agent applications.

The remainder of this article is organized as follows: Section II presents some low-level execution optimization methods. Section III describes resource allocation and parallelism mechanisms. Section IV discusses current FMs, as well as techniques for model compression and token reduction. Section V illustrates key components for agents. Section VI presents batching methods and some related applications. Finally, Section VII discusses the future works and draws conclusions.

Edge deployment of FMs poses severe challenges due to the heterogeneity of edge devices in terms of computing ability, hardware architecture, and communication bandwidth [16]. It is important to design computation optimization methods for different devices to accelerate model inference. Figure 3 provides an overview of this chapter. This diagram provides a global perspective for algorithm design (e.g., computation optimization). It also depicts the components of an edge computing system, including heterogeneous backends and the network. Table I shows some commonly used hardware devices at the edge. Traditionally, devices with CPUs have been deployed at the edge environment. However, due to their limited parallel computing capabilities and memory constraints, various specialized accelerators are designed for Deep Learning (DL) tasks. FPGAs have gained widespread attention for their programmable and parallel computing capabilities. ASICs, while non-programmable after manufacture, offer high speed and low power consumption. Many edge devices, such as personal computers and smartphones, have different computational resources, such as GPU and CPU. Consequently, many approaches focus on optimizing computational efficiency by jointly utilizing these resources for model inference. In-memory computing, with non-Von-Neumann architecture, has emerged as a prominent research area because its intrinsic parallelism can significantly reduce the I/O latency and improve computational efficiency. To enhance the understanding of different devices, we provide some detailed information about different hardware devices as below.

In addition to computation and memory optimization, communication overhead is another major challenge in deploying large models within the edge-cloud environment. The heterogeneity of edge devices and communication channels necessitates a scheduler for computation and network resources when executing FM tasks across different devices.

[59] introduces an “Intelligence-Endogenous Management Platform” for Computing and Network Convergence (CNC), which efficiently matches the supply and demand within a highly heterogeneous CNC environment. The CNC brain is prototyped using a deep reinforcement learning model. It theoretically comprises four key components: perception, scheduling, adaptation, and governance, collectively supporting the entire CNC lifecycle. Specifically, Perception perceives the incoming service request and real-time computing resources. Scheduling assigns the workload to heterogeneous computing nodes in a heterogeneous network. Adaptation is adapting to dynamic resources by ensuring the continuity of services through backup measures, and Governance is the self-governed decentralized computing nodes.

[60] discusses the integration of LLMs into 6G vehicular networks, focusing on the challenges and solutions related to computational demands and energy consumption. It proposes a framework where vehicles handle initial LLM computations locally and offloads more intensive tasks to Roadside Units (RSUs), leveraging edge computing and 6G networks’ capabilities. The authors formulate a multi-objective optimization framework that aims to minimize the cumulative cost incurred by vehicles and RSUs in processing computational tasks, including the cost associated with communication. LinguaLinked[61] is a system to deploy LLMs on distributed mobile devices. The high memory and computational demands of these models typically exceed the capabilities of a single mobile device. It utilized load balancing and ring topology to optimize the delay of computation and communication while maintaining the original model structure. The results demonstrate improvements in inference throughput on mobile devices. [62] presents MegaScale, a production system designed to train and deploy LLMs across over 10,000 GPUs. The synchronization of model parameters and gradients across GPUs can become a bottleneck as the number of GPUs increases. MegaScale addresses communication challenges through a combination of parallelism strategies, overlapping techniques, and network optimizations, resulting in improved training efficiency and fault tolerance [63].

Semantic communication enables much more efficient utilization of limited network resources at the edge by only transmitting the essential semantic information needed for communication purposes. [64] investigates multiple access designs to facilitate the coexistence of semantic and bit-based transmissions in future networks. The authors propose a heterogeneous semantic and bit communication framework where an access point simultaneously sends semantic and bit streams to a semantics-interested user and a bit-interested user. [65] presents a framework for semantic communications enabled by computing networks, aiming to provide sufficient computational resources for semantic processing and transmission. This framework leverages computing networks to support semantic communication. It introduces key techniques to optimize the network, such as semantic sampling and reconstruction, semantic-channel coding, and semantic-aware resource allocation and optimization based on cloud-edge-end computing coordination. Two use cases, an end-cloud computing-enabled video transmission system, and a semantic-aware task offloading system, are provided to demonstrate the advantages of the proposed framework.

With the emergence of computational and memory optimization methods, numerous integrated frameworks have integrated these techniques, supporting various LLMs and lowering the barriers to LLM deployment. These frameworks leverage CPU and GPU backends, enabling the widespread local execution of LLMs.

The most popular framework to deploy LLMs at the edge is the llama.cpp[66], which pioneered the open-source implementation of LLM execution using only C++. We provide an overview of various open-source frameworks and their respective features in Table II. The frameworks can be categorized into heterogeneous and GPU-only backends, ranging from mobile and embedded devices to high-end computing systems. Different suppliers offer specialized development platforms and APIs for their hardware products (i.e., CPU and GPU). CUDA (Compute Unified Device Architecture) is a parallel computing platform developed for NVIDIA GPU. ROCm (Radeon Open Compute platform) is a software platform for high-performance computing using AMD GPU. Metal is a low-overhead hardware-accelerated 3D graphic and compute shader API developed by Apple for its own GPUs. Some frameworks like Vulkan and OpenCL (Open Computing Language) facilitate development across different hardware and operating systems. Vulkan is an API for cross-platform access to GPUs for graphics and computing. OpenCL is a framework for writing programs that execute across heterogeneous backends, including CPU, GPU, FPGA, and other hardware.

Recently, several novel frameworks have been specifically designed for the deployment of LLM-based applications/agents. These frameworks provide abstract interfaces that facilitate the design of complex LLM-based applications, such as chain-of-thought reasoning and retrieval-augmented LLMs. LangChain is a powerful framework aimed at simplifying the development of applications that interact with LLMs [77]. It offers a set of tools and abstractions that enable developers to construct complex, dynamic workflows by chaining various operations, such as querying a language model, retrieving documents, or processing data. Parrot is a serving system for LLM applications that optimizes performance across multiple requests [78]. It introduces the concept of a semantic variable to define the input or output information of LLM requests. To enhance system performance, Parrot schedules requests based on a predefined execution graph and latency sensitivity, and it shares the key-value cache among requests to accelerate the prefilling process. SGLang is another LLM agent framework designed to efficiently execute complex language model programs [79]. It simplifies the programming of LLM applications by providing primitives for generation and parallelism control. Additionally, it accelerates the execution of these applications by reusing key-value caches, enabling faster constrained decoding and designing API speculative execution.

Edge-cloud computing leverages strong cloud servers and distributed edge devices to handle tasks near the data source. As shown in Figure 4, adaptive resource allocation is crucial for achieving an optimal balance between system performance and cost in edge-cloud environments. Resource management facilitates the optimal utilization of system resources, including processing power, storage space, and network bandwidth. Adaptive algorithms are designed to automatically adjust resource configurations based on real-time workloads and environmental changes, ensuring optimal performance under varying conditions.

Despite their numerous advantages, several challenges also arise in edge-cloud environments. 1) In edge-cloud environments, resources need to be allocated and managed between the central cloud and multiple edge nodes. Distributed resource management may increase the system’s complexity and require more fine-grained scheduling and coordination mechanisms. 2) The load in edge computing scenarios is highly dynamic. Accurately predicting this query load and dynamically adjusting resource allocation demands is difficult. 3) To meet the real-time processing requirements, the system must quickly adapt to changes in execution environments and adjust processing strategies and resource allocations in time. 4) Modern computing environments provide a variety of computing resources (CPU, GPU, TPU, etc.). Optimizing the allocation of these heterogeneous resources for various models based on their computational requirements and current workloads is a complex task.

We have summarized previous research works on resource allocation and adaptive optimization in Table III. The collected research works are divided into two categories: optimization for model services in cloud environments and optimization for IoT applications in edge computing environments. 1) Cloud Environments. This task is to efficiently deploy, manage, and scale machine learning models in the cloud. The objective is optimizing resource usage, reducing computing costs, and achieving performance goals in the face of dynamic and unpredictable query loads. This includes resource provisioning algorithms and strategies for utilizing heterogeneous computing resources. 2) Edge Environments. Because of the unique architectures and application scenarios in edge environments, data needs to be processed closer to the user or the geographic location of the data source. This requires well-designed scheduling algorithms and fault tolerance and resilience mechanisms to minimize latency and mitigate network overhead.

1) Cloud Environments. The following articles present various advanced solutions to optimize resource allocation in the cloud. Clipper uses model containers to encapsulate the model inference process in a Docker container [80]. Clipper supports replicating these model containers across the cluster to increase the system throughput and utilize additional hardware accelerators for serving. MArk is a generic inference system built on Amazon Web Services. It utilizes serverless functions to address service delays with horizontal and vertical scaling. Horizontal scaling expands the system by adding more hardware instances, while vertical scaling expands the system by increasing the resources of a single instance [81]. MArk uses a Long Short-Term Memory (LSTM) network for multi-step workload prediction. Leveraging the workload prediction results, MArk determines the instance type and quantity required to meet the SLOs using a heuristic approach. Nexus adopts the squishy bin packing method to batch different types of tasks on the same GPU, enhancing resource efficiency by considering the latency requirements and execution costs of each task [82]. It also merges multiple tasks into the same GPU execution cycle as long as the latency constraints are not violated. InferLine utilizes a low-frequency planner and a high-frequency tuner to manage the machine learning prediction pipeline effectively [83]. The low-frequency combinatorial planner finds the cost-optimal pipeline configuration under a given latency SLO. The high-frequency auto-scaling tuner monitors the dynamic request arrival pattern and adjusts the number of replicas for each model. Clockwork achieves an adaptive resource management framework through a fine-grained central controller over worker scheduling and resource management [84, 102]. DNN workers pre-allocate all GPU memory and divide it into three categories: workspace, I/O cache, and page cache. This can avoid repeated memory allocation calls and improve predictability. To cope with changes in query load, INFaaS adopts two automatic scaling mechanisms: vertical auto-scaling at the model level and horizontal auto-scaling at the Virtual Machine (VM) level [85]. The model auto-scaling is handled by the Model-Autoscaler, which decides each model variant’s scaling operations (replication, upgrade, or downgrade) by solving an integer linear programming (ILP) problem. The vertical auto-scaling adds a new VM if the utilization of any hardware resource exceeds a configurable threshold. Morphling utilizes model-agnostic meta-learning techniques to effectively navigate the high-dimensional configuration space, such as CPU cores, GPU memory, GPU timeshare, and GPU type [86]. It can significantly reduce the cost of configuration search and quickly find near-optimal configurations. Cocktail designs a resource controller to manage CPU and GPU instances in a cost-optimized manner and a load balancer to allocate queries to appropriate instances [87]. It also proposes an autoscaler that leverages predictive models to predict future request loads and dynamically adjusts the number of instances in each model pool based on the importance weight of the models. Kairos designs a query distribution mechanism to intelligently allocate queries of different batch sizes to different instances in order to maximize throughput [88]. Kairos transforms the query distribution problem into a minimum-cost bipartite matching problem and uses a heterogeneity coefficient to represent the relative importance of different types of instances. This allows for better balancing of the resource usage of different instance types. SHEPHERD comprises a planner (HERD), a request router, and a scheduler (FLEX) for each serving group [89]. HERD is responsible for partitioning the entire GPU cluster into multiple service groups. Besides, it performs periodic planning and informs each GPU worker of their designated service group and the models it must serve. SpotServe is a serverless LLM system that adjusts the GPU instances and updates the parallelism strategy flexibly [90]. It uses a bipartite graph matching algorithm (Kuhn-Munkres algorithm) to orchestrate model layers to hardware devices, thus maximizing the reusable model parameters and key-value caches.

2) Edge Environments. Given the limited and heterogeneous nature of edge computing resources, efficient management and scaling of these resources are crucial. The following papers present efficient solutions to improve the performance and efficiency of edge systems. To optimize resource allocation and interference management, [91] proposes a resource allocation scheme with Lagrangian and Karush-Kuhn-Tucker conditions. Based on the number of associated IoT end devices (IDs) and the queue length of the edge gateways (EGs), this scheme calculates the optimal resource allocation parameters and allocates resource blocks to each EG. A decentralized resource management technique is proposed in  [92] to deploy latency-sensitive IoT applications on edge devices. The key design is the deployment policy module, which is responsible for finding a task allocation scheme that meets the application’s requirements based on the bids from the bidder nodes and deploying the application to the corresponding edge nodes. The authors in  [93] introduce a novel method, called MDSA, to address the challenge of joint model partitioning and resource allocation for latency-sensitive applications in mobile edge clouds. They employ the task-oriented online scheduling method, COS, to collaboratively balance the workload across computing and network resources, thereby preventing excessive wait times. [94] designs a DRL algorithm to determine whether a task needs to be offloaded and to allocate computing resources efficiently. Tasks generated by mobile user equipment (UE) will be submitted to the task queue wait for execution. The algorithm models the task queue as a Poisson distribution. Then it uses the DRL method to select the suitable computing node for each task, learning the optimal strategy during the algorithm training process. Authors of [95] formulate the resource allocation problem using a Markov decision process model. They enhance the DQN algorithm by incorporating multiple replay memories to refine the training process. This modification involves using different replay memories to store experiences under different circumstances. The CE-IoT is an online cloud-edge resource provisioning framework based on delay-aware Lyapunov optimization to minimize operational costs while meeting latency requirements of requests [96]. The framework allows for online resource allocation decisions without prior knowledge of system statistics. The authors of [97] propose a dynamic optimization scheme that coordinates and allocates resources for multiple mobile devices in fog computing systems. They introduce a dynamic subcarrier allocation, power allocation, and computation offloading scheme to minimize the system execution cost through Lyapunov optimization. LaSS is a platform designed to run latency-sensitive serverless computations on edge resources [98]. LaSS employs model-driven strategies for resource allocation, auto-scaling, fair-share allocation, and resource reclamation to efficiently manage serverless functions at the edge. In [99], the authors discuss resource provisioning and allocation in FaaS edge-cloud environments, considering decentralized approaches to optimize CPU resource utilization and meet request deadlines. This paper design resource allocation and configuration algorithms with varying degrees of centralization and decentralization. KneeScale optimizes resource utilization for serverless functions by dynamically adjusting the number of function instances until reaching the knee point, where increasing resource allocation no longer provides significant performance benefits [100]. KneeScale utilizes the Kneedle algorithm to detect the knee points of functions through online monitoring and dynamic adjustment. CEC is a containerized edge computing framework that integrates workload prediction and resource pre-provisioning [101]. It adjusts the resource allocation of the container through the PRCT algorithm to achieve zero steady-state error between the actual response time and the ideal response time.

Since the release of ChatGPT, FMs, especially LLMs, have become increasingly important in daily life. Figure 7 presents a timeline of some popular LLMs. Table V highlights the structural features of several basic LLMs. Table VI and Table VII list various instruction-tuned models and multimodal models respectively.

Model adaptation in edge-cloud systems aims to dynamically select and possibly adapt ML models for inference tasks based on the current execution context, such as available computational resources, network conditions, and specific requirements of the task. Dynamic model selection and adaptation enable elastic acceleration in edge-cloud systems. By exploring the trade-off between performance, efficiency, privacy, and cost, this approach significantly enhances the feasibility and effectiveness of AI applications across different edge-cloud scenarios. While model adaptation for inference in edge-cloud systems brings substantial benefits, it also introduces several challenges. First, edge devices, such as IoT sensors or smartphones, often have limited computational power, memory, and energy resources. Adapting and running complex FMs within these constraints without compromising performance or accuracy is a significant challenge. Second, edge environments can be highly variable, with changes in network conditions, device capabilities, and application contexts. Dynamically adapting models to these changing conditions without human intervention requires real-time monitoring and long-term decision-making algorithms that can accurately assess the current environment and predict the best model or configuration. Third, quickly finding a sweet spot between inference speed (latency) and model accuracy is a challenge in different applications.

As shown in Figure 8, we categorize research works of model adaptation into three types. 1) Model Selection. The most prevalent method is model selection, which dynamically selects a suitable model for inference in a serving system. Research works design a model selector by considering various system characteristics. The first factor is hardware resources, which fundamentally determine the computational capacity of the system. Generally, lightweight models are deployed on edge devices, and stronger models are deployed on cloud centers. The second factor is the input. If the images are complex and belong to different domains, or if the language task is challenging to process, it is necessary to deploy a large model to enhance accuracy. The third factor is the service demands of users in different applications. Users’ requirements for accuracy and latency can vary significantly based on their use cases. For instance, those using entertainment devices like virtual reality may prioritize low-latency AI services, even if it means accepting a slight decrease in accuracy [242]. Conversely, users in the medical industry require impeccable accuracy, even if it results in longer waiting time for results [243]. The selector takes these factors as input and outputs the model selection strategy based on an optimization algorithm or a heuristic greedy method. A larger model can deliver more accurate results at the expense of increased latency, while a smaller model may provide faster responses but with potentially reduced accuracy. The model selector should find a sweet spot between latency and accuracy under different settings of a serving system. 2) Model Iteration. The model selection process should estimate both accuracy and latency in advance and then execute only one model per request. However, directly selecting a small model may lead to an inaccurate result. A few works execute model inference iteratively, starting from a small model and progressively moving to a larger model, and decide when to return results based on the prediction probability. The existing methods are designed based on the entropy of the probability, where a higher entropy value indicates increased uncertainty and necessitates forwarding the request to a larger model. 3) Speculative Decoding. Speculative decoding has emerged as an efficient and widely adopted technique in LLM inference to address the limitations of multi-step decoding. It utilizes a smaller model to generate a sequence of candidate words, which are then simultaneously verified by a larger model, resulting in improved performance. The benefits of speculative decoding stem from two aspects: (1) Small models can quickly generate text sequences, and (2) large models can group generated sequences into a batch to improve throughput by leveraging hardware resources more effectively.

We summarize recent works of model adaptation in Table XI. INFaaS is an automated model-less serving system, which generates model variants optimized along different dimensions and automatically selects the most appropriate variant for each query based on performance, cost, and accuracy objectives [85]. EdgeAdaptor is designed to efficiently manage the trade-offs between inference accuracy, latency, and resource costs for edge-based DNN inference services. This framework addresses the problem by jointly optimizing application configuration, DNN model selection, and edge resource provisioning dynamically, in response to fluctuating demand and system conditions [238]. JellyBean is a system designed for optimizing and serving machine learning inference workflows across heterogeneous computing infrastructures [114]. For each ML operator within a workflow, JellyBean selects a model that meets the accuracy requirements at the lowest possible cost. This process considers the interaction between models to estimate the overall workflow’s accuracy and utilizes a beam search algorithm to explore the space of possible model configurations efficiently. STI is an on-device inference system with two novel techniques: model sharding and elastic pipeline planning with a preload buffer [239]. STI manages model parameters as independently tunable shards, profiling their importance to accuracy and managing them on disk. Moreover, the elastic pipeline planning module utilizes a small preload buffer to initiate execution without delays, selecting and assembling shards according to their importance to maximize inference accuracy within resource constraints. Tabi is an inference system that employs a multi-level inference engine to serve queries using smaller models by default and only switches to more computationally expensive LLMs for more complex applications [240]. Tabi uses a calibrated confidence score to decide whether the results from smaller models are accurate or if a query should be rerouted to a larger model for processing. CATs trains additional prediction heads on intermediate layers of a transformer model and uses a meta consistency classifier to dynamically decide when to stop computation for each input, based on a unique extension of conformal prediction [241]. Leviathan et al. first introduce speculative decoding. Let $p(x_{t}|x_{<t})$ be the distribution of a target model $M_{p}$ and $q(x_{t}|x_{<t})$ be the distribution of a draft model $M_{q}$, outputs can be sampled from these two distributions. Then it samples from $q(x)$ and keeps it if $q(x)\leq p(x)$. If $q(x)>p(x)$, it rejects the sample with probability $1-\frac{p(x)}{q(x)}$ and samples $x$ again from an adjusted distribution $p^{\prime}(x)=norm(\max(0,p(x)-q(x)))$. LLMCad designs an on-device system with three novel techniques: constructing a token tree for broader candidate token pathways, a self-adjusting fallback strategy for error correction, and a speculative token generation during the verification process to maintain efficiency [52]. SpecInfer introduces a novel approach where small speculative models predict the LLM’s outputs, organizing these predictions into a token tree, with each node representing a candidate token sequence [51]. The system then verifies the correctness of all candidate sequences in parallel against the LLM, significantly reducing end-to-end latency and computational requirements while maintaining model quality. Sequoia employs a dynamic programming algorithm to determine the optimal speculative token tree structure, enabling it to scale with the size of the speculation budget [244]. Sequoia also features a hardware-aware tree optimizer that selects the optimal token tree size and depth based on the available resources to maximize speculative decoding performance. Minions employs multiple small speculative models to predict the output of an LLM, using a majority-voted approach to improve inference performance [245]. It also dynamically adjusts the speculation length of small models, optimizing the trade-off between the number of tokens speculated by small models and the verification cost by the LLM.

Although model adaptation can accelerate the inference of a transformer model, it may lead to large I/O latency because it needs to switch different versions of models between GPU memory and disk, which becomes the bottleneck during inference. By analyzing the inference process, researchers find that the significant computational cost of transformer models primarily comes at the self-attention mechanism [274]. It calculates the matrix multiplication between the key and query and scales quadratically with the sequence length. Processing long sequences or large batches of data can become computationally intensive because the number of tokens increases dramatically. Token reduction, encompassing token pruning, token merging and token summary, is an advanced technique in the field of artificial intelligence aimed at enhancing the efficiency and performance of large transformer models [275]. The core idea behind token reduction is to shorten the input data that a model processes, thereby reducing computational overhead and potentially improving the model’s ability to focus on the most important information. A key research topic in token reduction is to identify redundant or similar tokens in the input and remove or merge them without negatively affecting model performance.

Designing token reduction methods faces several challenges. 1) Information loss due to token reduction. Although token reduction can decrease computation costs, it often leads to significant information loss. The errors introduced by pruning strategies can adversely affect the model performance, as essential context and details necessary for accurate predictions may be discarded. 2) Trade-off between performance and efficiency. Achieving a good balance between model performance and computational efficiency is a major challenge. Aggressive token pruning can lead to large accuracy drops due to the loss of essential information. 3) Robustness to various reduction ratios. A large reduction ratio may inevitably remove crucial tokens, resulting in incomplete inputs. A robust token reduction method should maintain optimal performance across different reduction ratios. 4) Adaptability to different transformer architecture. The diversity of transformer architectures, including vanilla transformers and hybrid transformers, poses a challenge for developing a compression method that is both effective and flexible across different models. 5) Maintaining hardware-friendly inference. Another challenge lies in ensuring that reduction methods facilitate hardware-compatible inference, particularly when running on certain edge devices. This is crucial for the practical deployment of compression methods in real-world applications.

To solve the above challenges, a number of token reduction methods have been developed to accelerate the inference of the transformer. 1) Token pruning. There are only a limited number of words or image patches that contribute to the prediction of final results, and a lot of redundant tokens can be regarded as noisy information. Therefore, token pruning selectively removes tokens from the input sequence during the inference process of a transformer model. The core idea is to identify and retain only the most informative tokens for the task, thereby reducing the sequence length and the computational load. As shown in Figure 9a, there are two common approaches to indicate the significance of tokens that can guide the pruning process. The first method is training a prediction module, which takes tokens as input and outputs the importance score for each token. This module is constructed with a two-layer linear network and trained with soft masking of tokens. During inference, the $k$ tokens with the lowest importance scores will be removed to reduce the token number. The second method is directly collecting the attention weight to prioritize tokens. The attention mechanism calculates the interdependence among tokens, and the attention weight reflects the relationship between them. Tokens with lower attention values contribute less to the output of other tokens, therefore indicating their smaller significance. Therefore, the attention weight serves as a valuable metric for evaluating the token importance. 2) Token Merging. Token merging, on the other hand, combines adjacent or similar tokens into single tokens, effectively condensing the content and further reducing the sequence length without a substantial loss of information. This technique is particularly useful for large transformer models, where processing extensive sequences of tokens can be computationally intensive. The motivation behind token merging is the presence of numerous similar patches in images and videos, which exhibit similar functionalities within the transformer model. One of the most well-known methods is TOME, which demonstrates the effectiveness of token merging [265]. As shown in Figure 9b, the input tokens are initially divided into two sets, with tokens in set A selecting the most similar token in set B using cosine similarity of features. The top $k$ edges, representing the highest similarity of tokens, are retained. Finally, these tokens are merged using a weighted average approach to reduce the length of the input. 3) Token Summary. Token summary is a crucial technique for optimizing LLMs by condensing lengthy instructions and external knowledge sources. Instructions typically contain detailed task information and example references, while knowledge pools provide additional external information to assist the LLM’s inference process. However, these sources can be voluminous, potentially exceeding the model’s window size and causing latency issues. To address this, token summary expedites inference by summarizing the content into concise forms. It distills the content into smaller memory tokens using the LLM and concatenates them with user input tokens for inference. By leveraging token summaries, LLMs can effectively manage the overloaded information and enhance their ability to generate accurate responses.

Based on the application scenarios, we can classify existing token reduction methods into three types. We summarize all token reduction methods and list their contributions in Table XII.

1) Language: Token reduction is initially applied to the BERT model. Power-bert determines the pruned tokens based on the attention value and learns a configuration of how many tokens should be eliminated [256]. To improve the robustness, LengthDrop randomly generates a token length configuration during training and designs a multi-objective evolutionary search method to find an optimal token length during inference. Besides, a drop-and-restore process is employed to recover certain pruned tokens that might be important for the deeper layers [255]. AdapLeR trains a contribution predictor to evaluate tokens and design a soft-removal function to mask tokens for gradient propagation [254]. LTP assesses token importance with attention weight and learns a threshold to dynamically remove tokens [253]. To improve the accuracy of token importance evaluation, ToP transfers the token importance rankings derived from the final layer of unpruned models to the initial layers of pruned models [249]. They also introduce a coarse-to-fine pruning approach to dynamically select pruning layers. The above methods are designed for discriminative language models, such as Bert. In recent years, there has been a significant advancement in generative language models, such as GPT-3 and Llama. As a result, several studies propose token reduction techniques specifically tailored for these models. The self-information, such as entropy and perplexity, is used to measure the token importance for pruning [252]. LLMLingua uses a small model to compress prompts, sets different compression ratios for instructions, questions and demonstrations, and iteratively prunes fine-grained tokens to improve the accuracy [250]. A few methods aim to reduce the size of the Key-Value (KV) caches to expedite the decoding process. The authors of Heavy Hitter Oracle (H20) observe that a small subset of tokens, termed Heavy Hitters (H2), contribute most of the value when computing attention scores [248]. Based on this, they propose the H2O, a KV cache eviction policy based on the accumulated attention scores that dynamically retains a balance of recent and H2 tokens, effectively reducing the cache size without sacrificing performance. This method can improve the throughput by up to $29\times$ on OPT-6.7B and OPT-30B compared to other leading inference systems. FastGen incorporates four policies to dynamically adjust the KV cache [246]. These policies are as follows: 1. Retaining special tokens like $<$s$>$ and $<$INST$>$. 2. Preserving punctuation tokens like “.” and “?”. 3. Evicting tokens that are distantly located from the current token. 4. Retaining tokens with high attention scores. StreamingLLM is an efficient method for deploying LLMs in streaming applications, such as multi-round dialogue [247]. StreamingLLM leverages an observed phenomenon called “attention sink”, where maintaining the KV states of initial tokens significantly improves performance. This method can accelerate the inference of Lllama-2 by up to $22.2\times$ compared to other methods. For token summary, AutoCompressor and ICAE adopt compact summary vectors to distill information from the original contexts [259, 257]. Gist compresses instruction prompts into “gist” tokens that represent a specific task [258].

2) Vision. There are a lot of redundant tokens in an image, such as the background. DynamicViT introduces a trainable prediction module to determine the token scores. Paper [263] dynamically prunes less informative patches from the input image in a top-down manner by formulating token pruning as an optimization problem. HeatViT deploys attention-based token pruning methods on FPGA devices and also incorporates hardware optimization strategies such as 8-bit quantization [262]. Authors of METR find that the attention weight with class token ([CLS]) fails to gather task-specific information because the attention mechanism tends to concentrate on more general tokens at the initial layers [261]. This phenomenon may degrade the performance of attention-based token reduction because of the inaccurate token importance estimation. This paper proposes Multi-Exit Token Reduction (METR) that integrates a multi-exit architecture into ViTs to encourage the model to prioritize task-relevant information from the initial stages [261]. The motivation of STAR is to evaluate and prune patches based on their importance within and across layers [260]. It combines online intra-layer importance assessment with offline inter-layer importance analysis, using a fusion mechanism to selectively prune patches, aiming to retain those most critical for model performance. The idea of token merging originated from EViT [266], which divides tokens into attentive and inattentive tokens based on the attention weight and fuses inattention tokens into one token. TOME is a pure token merging method to gradually combine similar tokens within a transformer, achieving a balance of speed and accuracy that rivals pruning methods [265]. Recent works focus on combining token pruning and token merging to improve accuracy. Long et al. delve into the importance and diversity among image patches to preserve discriminative local tokens while maximizing global token diversity [268]. First, tokens are decoupled into two separate groups by leveraging class token attention. Inattentive tokens are then clustered using a density peak clustering algorithm, and attentive tokens are merged with a gentle matching method. TPS is also a token pruning and squeezing method [269]. First, it splits tokens into reserved and pruned subsets. Then, it employs unidirectional nearest-neighbor matching and similarity-based fusing steps to combine the information from pruned tokens into the reserved tokens. DiffRate enables automatic learning of different compression rates for different layers, thereby enhancing the optimization of the pruning and merging configuration [267].

3) Multi-modal & Video. When designing token reduction methods for multi-modal models, it is important to take into account the interactions between different modalities and adjust the strategy for evaluating token importance accordingly. In order to speed up the inference of video transformers, it is crucial to leverage both the redundancy within individual frames (intra-frame redundancy) and the redundancy between consecutive frames (inter-frame redundancy). TESTA extends the idea of token merging to video-language applications [271]. It begins by densely sampling input frames from the video. Subsequently, the framework incorporates two distinct merging schemes: frame merging, which combines similar frames, and token merging, which merges similar tokens. By employing these merging strategies, TESTA enhances the efficiency and effectiveness of video processing and analysis. PuMer employs a novel token reduction strategy that combines text-informed pruning and modality-aware merging techniques to selectively reduce the number of tokens from both input images and text [270]. It selectively removes image tokens that are irrelevant to the accompanying text and are deemed unimportant for the vision-language task predictions. STTS dynamically selects informative tokens in both temporal and spatial dimensions to improve the efficiency of video transformers [273]. By formulating token selection as a ranking problem, a lightweight scorer network estimates the importance of each token, and only those with top scores are used for downstream evaluation. Ding et al. introduce a method to optimize the speed-accuracy trade-off in video recognition tasks by pruning spatio-temporal tokens using a Semantic-aware Temporal Accumulation (STA) score. This score evaluates tokens based on temporal redundancy and semantic importance, allowing for the pruning of tokens that are temporally redundant or semantically less significant without the need for additional parameters or re-training.

Adaptive token reduction. Token reduction techniques demonstrate remarkable advantages in expediting transformer inference while maintaining prediction performance. Furthermore, it is crucial to develop adaptation methods for token reduction to cater to various applications across diverse execution environments. By doing so, we can fully leverage the potential of token reduction and optimize its effectiveness. OTAS incorporates token prompting and token reduction in an elastic serving system, enabling dynamic selection of an optimal token number [276]. This selection process takes into account factors such as fluctuating query load, diverse service targets, and limited hardware resources. It designs an optimization method to investigate the balance between accuracy and throughput across various token reduction ratios. AdaTape enhances the flexibility and performance of a transformer model by dynamically adjusting the computation based on the input’s complexity [277]. It employs an elastic input sequence mechanism through adaptive tape tokens (i.e., prompt), which are generated from a tape bank and appended to the input sequences, allowing for dynamic read-and-write operations. This method enables the model to adaptively control both the content and the number of tape tokens used for each input, thereby adjusting the computational budget and potentially improving efficiency and performance on tasks.

An LLM multi-agent framework is a system that leverages multiple LLMs as independent agents working collaboratively to handle complex tasks. Each agent may specialize in different areas or subtasks, and they communicate and cooperate to share information, verify outputs, and enhance overall performance. This approach improves robustness, efficiency, and scalability by enabling parallel processing and modular updates. For example, A multi-agent framework in software development organizes specialized roles such as architects, programmers, and testers into a cohesive, collaborative system. Each agent has distinct responsibilities and communicates effectively through defined protocols. This multi-agent framework enhances efficiency, reduces errors, and ensures consistency by leveraging task management systems, collaboration tools, and shared knowledge bases.

In multi-agent systems, communication and coordination are the foundations of achieving cooperation. It is necessary to design practical communication protocols to ensure that agents exchange information efficiently and accurately. Agents must maintain consistent semantic understanding and objectives to avoid misunderstandings and conflicts. The reasonable allocation and division of tasks are central to multi-agent cooperation. Tasks should be distributed among agents reasonably to avoid overloading some agents while leaving others idle. Knowledge sharing is the basis for achieving collaborative work in multi-agent systems. Different agents may have different backgrounds, and efficiently integrating this knowledge is challenging.

Multi-agent systems integrate the advantages of multiple LLMs with distinct functionalities. Each agent within these systems possesses expertise in specific domains, and this broad spectrum of specialized knowledge ensures that the generated results are comprehensive and accurate. Solving complex problems requires a multifaceted approach, and by leveraging the strengths of multiple agents, these systems can offer more refined and practical solutions. AgentVerse[279] is a multi-agent framework designed to facilitate collaborative problem-solving among autonomous agents powered by LLMs. AgentVerse emulates human group dynamics to improve task performance and explores emergent behaviors within agent collaborations. Agentbench[280] presents a comprehensive benchmark designed to evaluate the capabilities of LLM when acting as autonomous agents across a variety of real-world challenges. It consists of eight distinct environments categorized into code-grounded, game-grounded, and web-grounded scenarios to test the LLMs’ reasoning, decision-making abilities, and their performance in multi-turn, open-ended generation settings. CoELA[281] presents a novel framework for building cooperative embodied agents using LLM, focusing on multi-agent cooperation without requiring fine-tuning or few-shot prompting. This framework is evaluated in various embodied environments, demonstrating that agents can effectively plan, communicate, and cooperate on long-horizon tasks. The approach surpasses traditional planning methods and shows that natural language communication enhances cooperation with humans. LLM-Co explores the potential of LLM for multi-agent coordination, focusing on their ability to collaborate in various scenarios effectively[282]. CLIP[283] explores a novel approach to enhance the ability of LLM to execute tasks in embodied settings, such as with robots, where understanding the physical world is crucial. Semantic ICL offers a promising approach to improve conversational agents by leveraging both semantic search and LLM[284]. Yashar[285] introduces a novel framework to enhance the capabilities of LLM by leveraging multi-agent systems, allowing for collaborative environments where intelligent agents work to handle complex tasks efficiently. The authors demonstrate the framework’s superior performance by conducting case studies on models such as Auto-GPT, BabyAGI, and Gorilla, which incorporate external APIs into the LLM. DyLAN[286] aims to optimize the collaboration of LLM agents for complex tasks such as reasoning and code generation. DyLAN is introduced to enhance the performance and efficiency of LLM-agent collaboration on complex tasks by enabling dynamic interaction architecture and inference-time agent selection. Traditional methods employ a static ensemble of agents, limiting adaptability and requiring extensive human input for agent design. DyLAN overcomes these limitations through a framework that dynamically constructs agent teams based on task requirements, supports multi-round interactions among agents, and incorporates an automatic agent team optimization algorithm. This optimization relies on an unsupervised metric, the Agent Importance Score, to assess and select the most contributory agents. Empirical evaluations demonstrate DyLAN’s superiority in reasoning and code generation tasks, showcasing significant improvements over single-agent performances and traditional static approaches.

Multi-agent systems can fully leverage the expertise and skills of multiple independent agents to achieve solutions for complex tasks through collaborative cooperation. Each agent focuses on in-depth research in a specific domain, ensuring high professionalism, while the overall system integrates this information to provide comprehensive and accurate results. This architecture enhances the ability to handle complex problems. It possesses high flexibility and scalability, allowing adjustments and optimizations based on different needs, ultimately offering a more robust solution than a single LLM.

Intelligent agents enhance task-handling capabilities in complex systems by breaking down large tasks into smaller, more specific sub-goals. This decomposition strategy makes task management more feasible and improves overall efficiency and effectiveness. The planning has the following characteristics:

• •

  Hierarchical approach: Agents solve problems layer by layer by creating a hierarchical structure of tasks. From high-level strategic decisions to low-level specific operations, let LLM think step-by-step.

• •

  Parallel processing: After task decomposition, agents can process multiple sub-goals in parallel, utilizing resources effectively. For example, in multi-agent systems, different agents can simultaneously deal with various tasks.

• •

  Dynamic adjustment: Agents adjust the priority and resource allocation of sub-goals based on real-time feedback to adapt to environmental changes and unforeseen circumstances, ensuring the optimal task execution strategy.

Intelligent agents can enhance their ability to handle complex tasks through effective sub-goal setting, task decomposition, and continuous reflection and improvement. This strengthens the agents’ ability to solve problems independently and optimizes the entire system’s performance, contributing to efficient, adaptive, intelligent system design.

MindAgent[287] is designed to evaluate planning and coordination capabilities in gaming interactions. DEPS [288]is an innovative approach leveraging LLM to plan tasks in multi-task agents in open-world environments. DEPS uniquely integrates interactive planning with LLMs, focusing on error correction and efficiency improvement in plan execution through a goal selector module. It significantly improves task success rates in Minecraft and other tasks like ALFWorld and tabletop manipulation. MCTS[289] explores the integration of LLM with Monte Carlo Tree Search for task planning. This combination, termed LLM-MCTS, leverages the commonsense knowledge of LLMs to enhance the efficiency of planning algorithms. The key findings indicate that LLM-MCTS significantly outperforms traditional MCTS and LLM-induced policies in complex task scenarios, demonstrating the advantage of using LLMs as both a world model and a heuristic policy guide. ICPI[290] introduces a method for implementing policy iteration using LLM. ICPI updates the prompt content to derive policies through trial-and-error interaction with an RL environment. The experiments demonstrate the efficacy of this method across six RL tasks, employing a range of LLMs such as Codex and GPT variants. Code-LLMs[291] investigate the limitations of current code-oriented LLM in handling code completion tasks when the provided code contains potential bugs.

There are limitations to the direct use of LLMs as planners, such as their limited reasoning and planning capabilities and the inefficiency of the utilization of human feedback. PDDL[292] proposes a new method for planning tasks using pre-trained LLM. The method uses LLMs to construct PDDL models, employs PDDL validation and human feedback to correct initial errors in these models, and generates plans using the corrected PDDL models.[292]. ReCon[293] incorporates two processes: formulation contemplation for generating initial thoughts and speech and refinement contemplation for polishing these thoughts. MPC[294] proposes a novel approach for creating high-quality conversational agents without fine-tuning. This approach harnesses the power of pre-trained LLMs as discrete modules, thereby ensuring sustained coherence and adaptability in open-domain conversational contexts. The author demonstrates that MPC performs comparably with fine-tuned chatbot models in open-domain settings, offering an effective solution for generating consistent and engaging chatbots through human evaluations.

In an LLM-powered autonomous agent system, LLM functions as the agent’s brain. The agent breaks down large tasks into smaller, manageable subgoals, enabling efficient handling of complex tasks. The agent is capable of engaging in autonomous introspection and critiquing its past actions, extracting lessons from its errors, and subsequently refining its strategies for subsequent endeavors, culminating in an enhancement of the ultimate outcome’s caliber.

Based on the LLM, memory of the historical chat sequence is dynamically maintained and extracted through a retrieval method. The historical chatbot manages and updates a memory system to ensure relevant information is retained and can be accessed efficiently. This rethinking method helps refine, organize, and integrate this information for future use, enhancing the model’s ability to learn from past interactions and apply this knowledge to new situations. This methodology effectively improves the model’s decision-making capabilities and adapts to evolving contexts.

Many scholars are currently studying long-term memory in agents. Agents are capable of storing and retrieving information over extended periods, allowing them to maintain context and consistency across multiple interactions. REMEMBERER[295] introduces a novel framework for LLM that integrates a long-term experience memory, allowing the model to leverage past interaction experiences for decision-making across different tasks. This approach, termed reinforcement learning (RL) with experience memory, enables the LLM to evolve its capabilities without fine-tuning parameters, positioning it as a semi-parametric reinforcement learning agent. The methodology updates the experience memory through RL processes, and experiments on two RL task sets demonstrate REMEMBERER’s superiority over state-of-the-art methods. LONGMEM[296] enhances LLM by incorporating long-term memory capabilities. This framework addresses the limitations of current LLMs, which are constrained by a fixed-sized input, preventing them from leveraging extensive long-context information from past interactions. LONGMEM can handle unlimited-length context, significantly expanding its use cases, particularly in scenarios requiring understanding and generation based on extensive historical data. The model demonstrates superior performance on long-context modeling benchmarks such as ChapterBreak and shows remarkable improvements in memory-augmented in-context learning tasks over traditional LLMs. MaLP [297]is a novel framework for personalizing LLM like GPT-3.5 without fully retraining them, which is resource-intensive. The authors introduce a dual-process enhanced memory mechanism and a parameter-efficient fine-tuning schema, aiming to make LLMs more user-oriented by leveraging both short-term and long-term memory components in coordination. Hatalis[298] focuses on the agents’ memory management, particularly long-term memory, which is implemented using vector databases for storing and retrieving information. This aspect is crucial for maintaining context-specific knowledge and recalling past experiences, enabling coherent and efficient interactions. However, these methods require a lot of extra space to store long-term memory. To alleviate this issue, some research investigates the combination of self-reflection and memory mechanisms to achieve self-improvement without the need for additional data. MoT[299] aimed at enhancing the capabilities of LLM like ChatGPT without the need for additional annotated datasets or computationally expensive fine-tuning. This approach draws inspiration from the human ability to self-improve through self-reflection and memory. It proposes a mechanism for LLMs to self-improve by leveraging their internal generation processes and an external memory component. Liu[300] introduces Reasoning and Acting through Scratchpad and Examples (RAISE), a sophisticated architecture designed to improve the integration of LLM like GPT-4 into conversational agents. The architecture uses a memory system to maintain the continuity of contextual conversations better. It outlines a comprehensive method for the construction of agents, including historical conversation selection, scene extraction, chain of thought completion, and scene augmentation, leading to the LLMs training phase. The approach aims to enhance agent controllability and adaptability in complex, multi-turn dialogues. Maharana[301] introduces a machine-human pipeline to generate high-quality, very long-term dialogues leveraging LLM-based agent architectures. These dialogues are grounded on personas and temporal event graphs, and agents are equipped with the capability to share and react to images. Human annotators verify and edit the generated conversations for consistency and grounding to the event graphs. Wang[302] presents a novel approach to language modeling that incorporates corpus-level discourse information, termed a larger-context language model. This model introduces a late fusion technique to a recurrent language model based on LSTM units, which enables the model to distinguish between intra-sentence dependencies and inter-sentence dependencies more effectively. To improve the efficiency of retrieval-based generation, RaLMSpec and Speculative RAG are proposed to use a small knowledge pool to approximate a large knowledge base with a small knowledge base [303, 304].

Traditional LLMs remember patterns, facts, and information across the data they were trained on, allowing them to generate knowledgeable responses. With the support of the memory system, LLMs can quickly retrieve information from memory, making them highly effective for tasks that require rapid access to a broad range of information, such as question-answering systems or recommendation engines.

In the context of multi-agent systems, particularly those involving LLM, the capability for agents to call external APIs represents a significant enhancement to their functionality and adaptability. This approach addresses several limitations of pre-trained models, offering the following benefits:

• •

  Access to current information: LLMs, once trained, do not inherently update their knowledge unless retrained with new data. By enabling agents to call external APIs, they can access the most current information available, which is crucial for tasks that depend on up-to-date data such as news updates, stock prices, or weather forecasts.

• •

  Enhanced Functional Capabilities: External APIs can extend the abilities of an LLM beyond its original training. For instance, agents can perform calculations, access mapping services, or retrieve specialized data that is not stored within their pre-trained weights.

• •

  Interactivity with Proprietary Systems: Agents can interact with proprietary information sources that are not publicly available or part of their training data. This is particularly useful in enterprise settings where access to internal databases or specific industry-related systems is required.

• •

  Dynamic Adaptation to New Domains: The ability to call external APIs allows agents to dynamically adapt to new domains or changes in their operating environment without the need for retraining. This makes the system more flexible and responsive to user needs.

Many researchers design different automation and standardization methods for integrating diverse tools into the execution of LLMs, enabling LLMs to seamlessly invoke various APIs and external resources. They are dedicated to developing context-aware tool selection mechanisms that allow LLMs to intelligently choose the most suitable tools based on the dialogue content and achieve multi-step decision-making in complex tasks. Toolformer[305] is specifically designed to make informed decisions regarding the selection and invocation of appropriate Application Programming Interfaces (APIs). It determines not only the optimal timing for API calls but also the most suitable arguments to be passed to these interfaces. They integrate a diverse array of utilities, encompassing tools such as a computational calculator, a question-and-answer system, a robust search engine, a multilingual translation service, and a scheduling calendar. Remarkably, Toolformer demonstrates significantly enhanced performance in zero-shot scenarios across a wide array of downstream tasks, frequently matching or even outperforming much larger models. Gpt4tools[306] introduces an innovative methodology aimed at empowering LLMs, such as LLaMA and OPT, with the capacity to leverage tools. This is achieved by crafting an instruction adherence dataset through the strategic prompting of a sophisticated teacher model across diverse multimodal environments. Low-Rank Adaptation optimization techniques are then employed during the fine-tuning process, enabling these LLMs to undertake visual understanding tasks and generate images. The resultant system showcases marked enhancements in accuracy for both familiar and novel tool applications. ToolEmu[307] is a framework utilizing LLMs to emulate tool execution for testing LM agents against diverse tools and scenarios, identifying risks such as data leakage or financial losses. It features an LM-based automatic safety evaluator to quantify risks from agent failures. LLMs have shown remarkable abilities in various NLP tasks but struggle with issues like hallucination and numerical reasoning[308]. To enhance their capabilities, the integration of external tools has been explored. However, existing evaluation methods fail to clearly distinguish between LLMs leveraging their internal knowledge and those effectively using external tools. The ToolQA[308] dataset is introduced to address this gap, consisting of data from 8 domains and incorporating 13 types of tools for accessing external information. This dataset is unique in requiring the use of external tools for question answering, minimizing reliance on LLMs’ internal knowledge. Ruan[309] focuses on task planning and tool usage. They introduce two types of agents, one-step and sequential, to execute tasks through planning and using external tools. ToolkenGPT[310] enhances LLMs with external tools without the need for fine-tuning. This method aims to solve complex problems by incorporating a vast array of tools, such as calculators or databases. The key innovation of ToolkenGPT is representing each tool as a token and learning an embedding for it. This enables the LLM to call tools as easily as generating text, thereby combining the strengths of fine-tuning while overcoming their limitations. Automatic Reasoning and Tool-use[311] enhances the capabilities of LLMs by automatically generating multi-step reasoning chains and integrating external tool use. It generates reasoning programs for new tasks by retrieving related demonstrations and then uses external tools as needed within those programs. This approach allows for a structured and extensible way to incorporate complex reasoning and external knowledge into LLMs’ responses. INFERCEPT is a novel framework designed to optimize inference for augmented LLMs that interact with external tools or environments [312]. Current LLM inference systems handle external interactions by discarding the context and re-initiating a new request after receiving the response, leading to significant GPU resource wastage due to the recomputation of context. INFERCEPT addresses this by minimizing GPU memory waste through improved handling of intercepted LLM generations, employing techniques like swap pipelining and recomputation chunking. It dynamically selects a strategy from discarding, serving, and swapping. The framework significantly improves throughput and request completion rates.

Using APIs, agents can leverage external computing resources, which can be more efficient and scalable than processing everything locally. This is especially valuable for resource-intensive tasks. This approach transforms the agent from a static information processor to a dynamic information seeker, greatly enhancing its utility and ensuring its relevance regardless of when it was last trained. This capability is central to developing intelligent systems that are expected to operate in real-world, ever-changing environments.

Batching is a widely adopted and beneficial strategy in serving systems, as it involves grouping incoming queries into batches and executing inferences for them together. This approach offers several advantages, including improved throughput and reduced resource consumption. The advantages of batching can be attributed to two primary factors. Firstly, it helps mitigate the costs associated with RPC (Remote Procedure Call) calls and minimizes overhead in internal frameworks, such as the need to copy inputs to GPU memory. Secondly, batching enables machine learning frameworks to leverage data-parallel optimizations more effectively. By performing simultaneous batch inference on multiple inputs, the framework can exploit parallel processing capabilities, such as those offered by GPUs, to enhance overall inference performance.

However, there are certain challenges associated with implementing batching in a serving system. We summarize the issues and their corresponding methods in Table XIII. First, when processing text queries, requests usually have different lengths. To handle this, padding with zeros is often used for shorter requests. However, this approach can result in computational inefficiency when there are significant differences in request lengths. Additionally, requests can also have different output lengths, ranging from just a few words to long articles. In such cases, shorter requests may need to wait for longer requests before returning results. TurboTransformer addresses this issue by employing a dynamic programming algorithm that groups requests with similar input lengths [316]. By doing so, it maximizes the overall throughput and reduces the inference latency. PiA takes into account both the input length and the output length of requests [315]. It introduces a predictor that accurately perceives and predicts the response length. PiA groups similar requests together, allowing for better resource allocation and improved efficiency. Dvabatch proposes a multi-entry multi-exit batching scheme, utilizing three meta operations—new, stretch, and split to dynamically adjust ongoing batches for different types of diversities. This approach allows for more flexible and efficient processing, such as enabling short queries to exit early, splitting batches to match an operator’s preferred batch size, and allowing later queries to join ongoing batches. [314]. ORCA introduces a novel scheduling mechanism called iteration-level scheduling, which allows for more efficient processing of inference requests by scheduling execution at the granularity of iteration rather than the entire request [313].

The second challenge involves addressing various service features of requests, including arrival time, latency requirements, and utility. For instance, one common dilemma is balancing throughput and latency, as prioritizing high throughput may lead to longer wait times for individual requests, potentially violating latency requirements. Clipper tackles this issue by employing dynamic adjustments to the batch size and modify the delayed batching [80]. This approach allows more queries to be batched together, enhancing throughput, while still considering the latency requirements of individual queries. Similarly, OTAS designs its batching algorithm by taking into account similar service characteristics [276]. This ensures that requests with comparable attributes are grouped together, optimizing the overall system performance. The premise of batching is that requests are processed by the same model. However, pre-trained models are fine-tuned with LORA for different tasks, resulting in distinct model parameters tailored for specific queries [320]. To deal with this issue, FLORA introduces an example-specific adapter [317]. This approach is computationally efficient, as it leverages matrix multiplications and element-wise operations that are inherently batch-friendly on modern GPUs. S-LoRA and Punica design some highly optimized CUDA kernels for heterogeneous batching of LoRA computation [318, 319].

Generative AI has demonstrated robust performance and vast potential in commercial applications, playing a significant role across various industries. Commercial generative AI can be categorized into four types: (1) text-generating AI (including some multimodal AI), (2) code-generating AI, (3) image-generating AI, and (4) video-generating AI. Among these, text-generating, code-generating AI, and image-generating AI have already seen widespread application. Video-generating AI slightly lags behind the former three, but it exhibits considerable potential for future application.

• •

  Text-generating and multimodal AI. ChatGPT is the most famous chatbot developed by OpenAI, and performs well in various NLP tasks[321]. ERNIE Bot (Wenxin Yiyan) is similar to ChatGPT, supporting input and output of images and text, and outperforms ChatGPT 4.0 in several Chinese tasks[322].

• •

  Code-generating AI. Github Copilot is the most widely adopted code-generating tool in the world. It can automatically generate code and comments, and provide code explanations, allowing developers to focus on problem-solving and collaboration[323]. Codex, similar to GitHub Copilot, is a product based on GPT-3 and is proficient in lots of programming languages. OpenAI claims that Codex is more powerful than GitHub Copilot[324].

• •

  Photo and Video-generating AI. Midjourney can generate high-quality images according to natural language. It entered public beta on July 12, 2022, and is continuously evolving[325]. Sora, based on diffusion models, GPT and DALL·E, can generate videos from images and language, or expand on already generated videos. OpenAI claims that it is an important step towards AGI[326]. Runway Gen-2 is a high-performance video-generation AI developed by Runway AI, which can take text, image, or video input and convert it into new videos[327].

The rise of LLMs has significantly accelerated research related to AI agents, which are now considered a principal avenue toward achieving AGI. Andrej Karpathy, co-founder of OpenAI, has stated that AI agents represent a significant future direction for AI. In the development of digital entities across various industries, these agents are expected to play a pivotal role in applying AGI. AI agents, as products, are anticipated to conduct business operations. Since March 2023, the field of AI Agents has witnessed several groundbreaking research developments. AutoGPT, a project developed by Significant Gravitas, automatically optimizes GPT’s hyperparameters by searching for algorithms that enhance its performance across diverse tasks [328]. Users simply set one or more objectives, and AutoGPT can autonomously generate and execute tasks, continually analyzing and refining its strategies throughout the process. Researchers at Stanford and Google have developed a simulated environment named Smallville, where 25 AI agents exhibit complex behaviors and interactions [329]. Each agent possesses a unique seed memory, which enables them to form and recall relationships through social interactions with other agents. VOYAGER, designed with the long-term goal of ‘discovering as many different things as possible’, is the first AI agent in Minecraft to continuously explore the world, acquiring a diverse set of skills and making new discoveries [330]. It introduces an expanding arsenal of skills for storing and retrieving complex behavioral executables, alongside a novel iterative cueing mechanism that enables the agent to autonomously explore and adapt to unknown environments without human intervention, demonstrating superior proficiency. ChatDev (Chat-powered Software Development) is proposed as a virtual software company operated by multiple intelligent entities [331]. After a user specifies a task, ChatDev utilizes the software engineering waterfall model. Through a sequence known as the Chat Chain, consisting of atomic tasks, it facilitates automated interactions, collaboration, and decision-making among diverse AI agents to produce comprehensive software, including source code, environment dependency specifications, and user manuals. Research on AI Agents is currently led primarily by the academic community and developers. The outstanding performance of AI Agents in independent operation, collective collaboration, and human-machine interaction is increasingly recognized as a potent tool for enhancing productivity across various industries in the digital economy. However, the widespread application of LLM-based agents faces significant challenges due to the costs associated with token interactions. Additionally, the absence of a profit-sharing mechanism indirectly hampers the development of AI Agents. The main strategies for reducing costs include combining different model sizes and optimizing inference infrastructure. Furthermore, rapid advancements in hardware and models are expected to resolve cost issues in the near future. OpenAI predicts that AI will surpass human intelligence levels within the next decade, achieving what is termed superintelligence. AI Agents are expected to become mainstream in future products, with numerous agent-centered products likely to emerge and be implemented across various fields in the coming years.

FM-powered agent services are widely recognized as a crucial way of achieving AGI and are expected to spearhead development in this field over the next decade. Constructing a robust infrastructure for these agent services within an edge-cloud environment is of great importance and has a substantial impact on the user experience. In this survey, we have presented a unified framework for a thorough literature review on diverse techniques for deploying FM-powered agent services. We have introduced a series of low-level optimization methods for the model execution, including computation, communication, and I/O optimization. Subsequently, we have discussed the parallelism methods and resource allocation schemes designed to optimize resource utilization. We’ve also highlighted several popular FMs and introduced two lightweight methods, namely model compression and token reduction, to expedite the inference processes. Furthermore, we have also reviewed the latest research endeavors focusing on the development of an effective multi-agent framework and explored several recent applications. Finally, the future research directions for serving multi-modal agents on heterogeneous devices are outlined.

This survey is anticipated to significantly advance the development of large-scale model applications in both academia and industry. The proposed framework, along with the referenced research works, provides a comprehensive perspective on deploying FM-powered agent services. It showcases the latest advancements and encourages more researchers to contribute to this practical and compelling field. The technologies discussed in the paper collectively contribute to the development of a reliable and flexible serving system, enabling low-latency agent services that enhance users’ daily experiences with increased intelligence. The integrated optimization of system architecture and AI algorithms will expedite the deployment of large-scale model applications for a diverse range of users, thereby promoting societal progress. Looking ahead, we are dedicated to advancing research in large model applications and aim to expand our investigation to include a broader range of models, execution environments, and practical applications.