Towards Fast Setup and High Throughput of GPU Serverless Computing

GPU container toolkit [45] is often used to integrate GPU into existing serverless frameworks seamlessly. The GPU calls of functions are forwarded to the underlying GPU driver by the GPU container toolkit. The container toolkit also provides the memory and computing resource partition to meet the multi-tenant requirements of GPUs in serverless computing. Both GPU-enabled Azure Functions [13] and Alibaba Function Compute [15] use the toolkit. With the help of the GPU container toolkit, they are designed with the FixedGSL rules for the convenience of management. That is, each GPU function is wrapped into the container equipped with GPU instances. These GPU instances have limited specifications, and the memory allocation granularity is 1 Gigabyte.

Considering that GPU serverless is suitable for a variety of fields, we also use multiple scientific computing tasks as the benchmarks in addition to the DL inference tasks. We use seven DNN models concerned by the DL inference system [46] and three scientific tasks from Parboil [2] as our GPU function benchmarks. Table 1 presents the benchmark details. The DL models cover computer vision, speech recognition, and natural language processing. The scientific computing tasks include fluid mechanics ($lbm$), medical assistance ($mrif$), and astrophysics ($tpacf$).

We run the benchmarks on a node with one NVIDIA A100 (40GB memory) GPU and use FixedGSL to manage the GPU resources. Upon each new request, FixedGSL allocates memory for the function in 1 Gigabyte granularity. Specifically, the function’s memory usage is rounded up from the real memory usage. The detailed hardware and software configurations are described in Table 3 of § 7.1.

In this subsection, we measure the response latencies of the benchmarks, and the system throughput with FixedGSL. The experiments show that FixedGSL results in long response latency and low system throughput. The long response latency comes from the extra setup stages for preparing the GPU functions. The low function density is attributed to the resource contention in the data preparation stages.

Figure 5 shows the overview of SAGE and its corresponding programming interface. As shown in the right part of Figure 5, SAGE comprises four modules: the per-function engine, the taxon shim, the unified memory daemon, and the kernel executor. SAGE proposes two innovative mechanisms to enhance the function latency and throughput performance for the GPU serverless system, which are parallelized function setup and sharing-based memory management.

When the upper cluster scheduler assigns a GPU function request to SAGE runtime, SAGE first starts a container with GPU capability. Inside the container, a per-function engine is then initialized. The engine first extracts the data that can be loaded in advance and notifies the memory daemon to perform data preparation. After that, the engine continues to trigger the execution of the function. Because the memory daemon works independently, the data preparation and the context preparation inside the engine are parallelized.

When the function handler is being executed, taxon shim intercepts all GPU calls and dispatches them respectively. Specifically, if it is a memory-related call, the shim forwards it to the memory daemon. The memory daemon returns the actual device memory address of preloaded data to the function handler. GPU Kernel-emitting calls are forwarded by the shim to the kernel executor. The kernel executor then communicates with the memory daemon to ensure that related GPU memory is correctly prepared.

SAGE deploys only one memory daemon on a single GPU. The memory daemon and the function engine collaboratively support the sharing-based memory management mechanism. When multiple invocations of the same function are scheduled to the same GPU, the read-only data will be only loaded upon the first invocation. The other invocations could share the same copy, which reduces the data loading requirement and the resource contention in the data loading paths. As a consequence, the function throughput is improved.

In addition, the per-function engine is also shared with the latter invocations. SAGE keeps the engine alive within a period. To enhance the sharing efficiency, the lifecycle of the launched per-function engine ends with a multi-stage exit method, rather than terminating directly upon the completion of the invocation’s execution. Specifically, the occupied resources of a function engine are released in multiple stages according to the order of read-only data, GPU context, CPU context, and container.

The left part of Figure 5 presents an example of SAGE’s programming interface. Our current implementation of SAGE only supports C++ language runtime. It should be trivial to provide other language bindings (e.g., Python), as GPU relies on C++ to provide its functionality. The provided interfaces follow the design pattern of the existing serverless framework AWS Lambda [47] that supports C++ runtime.

The developer wraps the GPU function code within the function handler with a pre-defined signature. SAGE also requires developers to store metadata of the data loaded from external sources when defining Request struct, which is detailed in § 5.1. The read-write attribute of data also needs to be designated to ascertain whether it can be shared as read-only data. SAGE’s runtime can then identify them for setup parallelization and memory sharing.

SageInit() and SageRun() are two APIs provided to initialize the SAGE’s runtime and invoke the function execution upon an incoming request. These two APIs do not need user definition. Later, we will present the interface details about how SAGE parallelizes the context creation and data preparation. Besides, we will also give an example of how to utilize our interface to transform the original GPU application into a GPU function for SAGE.

We outline the simplified execution flow of the GPU functions in Figure 6, which consists of seven stages. (1) The GPU scheduler creates a container for the GPU function. (2) The GPU function establishes its own CPU context and corresponding function engine. (3) The GPU function requests CPU-side memory and loads data from a designated database based on the invocation request. (4) The GPU function generates its own GPU context and initializes the corresponding library. (5) The GPU function requests GPU memory and transfers the data from CPU to GPU. (6) The GPU function starts computing. (7) Upon completion of computing, the GPU function returns the result.

Among these stages, stage-3 and stage-4 could be executed in any order for no dependency. Meanwhile, stage-5 and stage-6 can also occur in any order because the GPU memory allocation and the GPU kernel launch could happen at any time. However, developers usually prioritize preparing the data first before launching a series of GPU kernels. In this way, kernel launching overhead is hidden, and GPU utilization is improved. This means that the first five stages are executed sequentially prior to the computation stage. As a result, the GPU function experiences a long setup.

After a deeper analysis of Figure 6, we have two observations. First, GPU functions always load data from external sources. For instance, DL inference services commonly read model inputs and weights from databases, while scientific computing services require input data or updated data to be sourced from databases.

Secondly, both CPU and GPU support inter-process communication. CPU and GPU could use data from other processes for computation. Corresponding experimental results show that a GPU function can compute correct results through the exclusive use of data from other processes. This means that context and data are not firmly intertwined.

Based on these two observations, we have a key insight: GPU data preparation can be delegated to another process, enabling parallel preparation of both the GPU data and GPU context. Further, we reconsider the execution flow of a GPU function and have three findings. (1) Container creation, CPU context creation, and GPU context creation must be performed in a strict sequence. (2) There is no strong bond between the GPU data and GPU context, and GPU data loading depends on CPU data loading. (3) GPU computation depends on both GPU context and GPU data.

We reconstruct the dependency graph for the GPU function setup stages. As shown in Figure 6, we could divide the function setup stages into two distinct parts, namely GPU context preparation and GPU data preparation. The function engine is designated with GPU context preparation, whereas the memory daemon handles GPU data preparation from external sources. They are executed in parallel.

To implement the aforementioned process, the function engine must communicate the data-loading information with the memory daemon ahead of actual execution. To this end, we devise a data structure to store the positional details of the external data required by a request, as illustrated in Figure 8.

Specifically, Request contains all the data that require loading from the databases. The database information is stored in the key of the Data structure. When the memory daemon receives a request from the function engine, it immediately initiates the data-loading process. After the data loading is finished, the data_dptr could used to perform the computation. Leveraging this approach, GPU context preparation and GPU data preparation can occur concurrently, thereby enabling the fast function setup.

Loading data proactively through the unified memory daemon can invalidate the execution of the original GPU program. Previously the program interacts with the GPU directly, the GPU’s built-in scheduler ensures the correctness of memory operation, kernel launching, etc. With the unified memory daemon, there has to be a consolidated design to achieve exactly the same functionality. Then, the taxon shim is designed in response to the particular need.

To reduce resource contention in the data loading paths, we initially conduct experiments to investigate the memory usage of mainstream GPU functions. Our investigation reveals that GPU function memory usage can be divided into three categories: context memory, read-only memory, and writeable memory. Context memory is implicitly allocated during the initial execution of the GPU application and is utilized to store the runtime data. Read-only memory represents the read-only data of GPU function execution. Writeable memory, on the other hand, denotes the writable data of GPU function execution.

Table 2 demonstrates the memory usage allocation of all GPU functions. These functions are implemented by Rammer [46] and Parboil [2]. As observed from the table, context memory occupies the maximum memory usage of all functions except Bert, accounting for an average of 79.0% of the overall memory usage. Although context memory occupies a large amount of memory, corresponding experimental results show that the context creation time does not change when multiple function invocations create their contexts. This implies that context creation for function invocations does not interfere with each other.

In addition to context memory, explicit memory accounts for 21.0% of the overall memory usage. As demonstrated in § 3.2.2, concurrent function invocations encounter severe resource contention during data loading. The benchmark functions suffer 34.9$\times$ data loading time compared to the solo-run case. Meanwhile, we observe from this table that read-only memory may occupy more than 90% of explicit memory usage. Optimizing the data loading process for read-only memory can substantially mitigate memory contention issues in data loading paths.

Existing GPUs only offer basic data transfer APIs through PCIe, which could only handle data transfer tasks sequentially. Meanwhile, the disk and network bandwidth used by CPU loading are also limited. There lack sophisticated interfaces to manage the data transmission of multiple invocations to maximize the efficiency of data loading. Under these circumstances, we find that multiple invocations of the same function can safely share the read-only memory.

While read-only memory remains unmodified during the function execution, the read-only memory sharing between multiple invocations does not cause correctness issues or performance problems. It should be noted that multiple function invocations perform the same computation, which comes from the same service provider. There will also be no malicious memory attack on the GPU. In this case, read-only memory sharing can effectively reduce data loading, which further reduces the resource contention in the data loading paths. Therefore, the overall throughput can be improved.

To support the read-only memory sharing between multiple invocations, the memory daemon needs to identify read-only data in the function. Since function developers could easily distinguish read-only data from writable data, we require the function developers to indicate the data attributes in the data structure shown in LABEL:fig:request. If a data is read-only, the Type in the Data structure should be set to ReadOnly. Otherwise, it should be set to Writable.

It is worth noting that read-only memory is not limited to the weight of ML inference tasks. The $tpacf$ benchmark in astrophysics also has read-only memory. Any read-only data in the GPU function could be shared by multiple invocations.

Another commonly used sharing method involves reusing launched containers and keeping them warm for periods of time. Although the initial function call must complete all necessary prerequisite setups (referred to as a cold start), subsequent calls may be processed by the warm container, thereby mitigating undesired startup latency. Leading serverless platforms, including OpenWhisk [48] and AWS Lambda [49], mostly offer support for container reuse.

The warmed container exempts the invocation from some setup stages, which include container creation, context creation, and data loading. However, simply applying the warm strategy to GPU functions may not be a good choice. Specifically, maintaining a warm container for extended periods requires extra memory usage, such as GPU context memory and read-only memory. While GPU memory is scarce, maintaining a warm container for GPU function may cause a decrease in function density.

To solve this problem, we leverage the CPU-side context as a suitable intermediary layer for GPU functions. Given that multiple prerequisite stages are required for the computation of the GPU function, we design a multi-stage resource exit mechanism. Figure 9 shows our multi-stage exit mechanism.

In the first stage, the GPU function retains all GPU context and read-only data. In the second stage, the GPU function preserves the GPU context, yet caches the read-only data to CPU memory. This is because CPU memory typically boasts larger storage capacity, lower cost, and faster PCIe transfer times. In the third stage, the GPU function discards the GPU context but still retains the CPU memory and CPU context. In the fourth stage, the GPU function discards the CPU memory and CPU context but retains the container to minimize the start time of subsequent requests. Finally, after the container has served its purpose, it is destroyed.

Leveraging the multi-stage exit mechanism outlined above, we are able to attain a more flexible management of the tradeoff between GPU resource cost and warm containers. Moreover, it is noteworthy that the time interval of each stage is adjustable. For instance, configuring each stage’s interval to the previous time interval would enable the GPU function to maintain a longer warm state at a lower cost. Alternatively, configuring the overall duration of the three stages to the previous time interval would allow the serverless platform to support a similar warm effect at a lesser cost. In this paper, SAGE configures each stage’s interval to the previous time interval, which is set as 30 seconds[14, 18].

Note that, while context memory sharing does not mitigate the resource contention in the data loading paths, it still brings considerable context creation time and memory usage. Therefore, we also support the context memory sharing between multiple invocations of the same function. Using the context memory sharing, the function invocations could benefit from the shorter function setup and less memory usage. The shorter function setup further improves the system throughput, and the less memory usage enables the GPU to keep more functions warm on the GPU. Similarly, the invocations of the same function perform the same computation, there will be no malicious attack about the context.

We implement SAGE with all components described in § 4. Our system uses Docker [50] for the application sandbox, GRPC [51] for the message passing, and MongoDB [52] for the database. All components of SAGE are implemented as independent processes based on C++.

We use FixedGSL, FixedGSL-F, and DGSF as the baselines. FixedGSL is only capable of allocating memory for functions at a granularity of 1 Gigabyte. We extend FixedGSL to FixedGSL-F, which supports flexible memory allocation for functions. Additionally, we also choose DGSF, a state-of-the-art serverless system, as another baseline. Each function in DGSF has four pre-created GPU contexts, eliminating GPU context preparation time. Invocations to the same function share these four contexts using a First-Come-First-Serve queue [28]. Note that, to better distinguish our effect from previous works on the CPU side [16, 17, 18], we enhance all the systems with the pre-warmed container. Therefore, there is no container creation overhead for GPU functions.

To evaluate the effectiveness of SAGE, we use seven DNN applications from Rammer [46] and three scientific computing applications from Parboil [2] as our benchmark suite. These applications cover diverse domains such as object recognition, speech generation, natural language processing, fluid mechanics, and real-time medical imaging. Detailed information about the benchmarks can be found in Table 1 of § 3. Besides, we replay the workload trace from Microsoft Azure Functions (MAF)  [53] for 2 hours in all experiments. After we adapt the benchmarks to GPU functions using our interfaces, we could conduct the experiments.

The experiments described in § 7.2 to § 7.7 are executed on the Nvidia A100 GPU. All the experiment setups are shown in Table 3. First, the experiments in § 7.2 demonstrate the function latency performance and the system throughput performance of SAGE. Second, the experiments in § 7.3 are performed to compare with the theoretical performance. Third, the experiments in § 7.4 prove the performance of SAGE with more functions. Then, we conduct the ablation study for the multi-stage resource exit mechanism and read-only memory sharing scheme in § 7.6 and § 7.7. Finally, SAGE is deployed on a small cluster containing 4 Nvidia A100 GPUs to prove the scalability in § 7.8.

Figure 10 illustrates the average latency of ten applications for FixedGSL, FixedGSL-F, DGSF, and SAGE. On average, SAGE outperforms FixedGSL, FixedGSL-F, and DGSF by 193.4$\times$, 391.5$\times$, and 13.3$\times$, respectively. SAGE outperforms FixedGSL, FixedGSL-F, and DGSF on the minimum by 26.3$\times$, 52.4$\times$, and 1.21$\times$, respectively. Besides, we also collect the 99%-ile latency of ten applications, which are not shown due to page limitation. For the 99%-ile latency, SAGE outperforms FixedGSL, FixedGSL-F, and DGSF by 54.1$\times$, 109.2$\times$, and 25.4$\times$, respectively.

The improved latency performance of SAGE comes from two aspects. Firstly, SAGE accelerates the processing of cold invocations by paralleling its context preparation and data preparation. Secondly, sharing read-only memory and existing context also shortens the end-to-end latency of warm (invocations to the same GPU function).

As a comparison, FixedGSL and FixedGSL-F do not prioritize the latency performance of functions. They only schedule the function invocations with the memory usage in a fixed granularity. Although multiple invocations from the same function in DGSF could enjoy the pre-created GPU contexts, DGSF ignores the resource contention in the data loading path, thus still suffering long data preparation. Our proposed method, SAGE, takes both of these factors into account and has been shown to be highly effective in reducing latency.

Figure 11 illustrates the system throughput of FixedGSL, FixedGSL-F, DGSF, and SAGE during this period. SAGE’s system throughput outperforms FixedGSL, FixedGSL-F, and DGSF by 8.9$\times$, 10.3$\times$, and 1.22$\times$, respectively. The throughput improvement comes from the memory sharing of SAGE. The read-only memory sharing between multiple invocations could mitigate the resource contention in the data preparation stages. Thus, all the invocations could finish the computation in a short time, which contributes to higher system throughput.

It should be noted that FixedGSL-F shows worse performance compared with FixedGSL both in latency and throughput. This is because FixedGSL does not concern the resource contention in the data preparation stages. While FixedGSL-F could launch more function invocations on the GPU, it suffers more serious and unrestrained contention of the data-loading paths than FixedGSL.

Figure 12 illustrates the memory usage of FixedGSL, FixedGSL-F, DGSF, and SAGE during this period. The average memory usage of SAGE only accounts for 18.7% of FixedGSL, 21.7% of FixedGSL-F, and 37.5% of DGSF. The improved memory usage also comes from the memory sharing of SAGE. SAGE could achieve better latency and throughput performance using at most 37.5% memory of the baseline systems. Note that while SAGE uses less memory, it could hold more cold functions on the GPU, thus further improving the function latency performance.

In this section, we use the same experimental setup as § 3 to evaluate the theoretical throughput ratio for SAGE. The gray bars in Figure 13 shows the throughput ratio of each application using SAGE. On average, the system throughput is 65.1% of its theoretical value. These results demonstrate that SAGE has better performance than the baseline systems. The improved performance of SAGE comes from the fast setup and memory sharing of function invocations.

The red bars in Figure 13 show the normalized reduced time of each application using SAGE compared to the solo-run case. SAGE reduces the setup time of function invocations by 66.8% on average. While the function invocation could get the data ready in a short period, the system throughput could be improved. Note that, there is a significant gap between SAGE and the theoretical value because of the resource contention. Although SAGE could reduce the data loading pressure by sharing the read-only memory in the GPU function, some data preparation could not be avoided. Furthermore, some tasks like $slbm$ could not benefit from the read-only memory sharing.

In this subsection, we use 30 functions to further demonstrate the effectiveness of SAGE. While the system is required to handle more functions, the memory-sharing capability between multiple invocations of the same function is reduced. Specifically, we use 10 benchmarks in Section 7.1 to simulate 30 benchmarks. For example, we use $bert1$, $bert2$, $bert3$ to represent different benchmarks.

Figure 14 shows the average latency of 30 functions under FixedGSL, DGSF, and SAGE. On average, SAGE outperforms FixedGSL and DGSF by 211.9$\times$ and 5.9$\times$, respectively. Additionally, SAGE outperforms FixedGSL and DGSF with the tail latency by 49.6$\times$ and 4.6$\times$, respectively. While more functions on one GPU node mean a low load for each function, the memory-sharing opportunity between multiple invocations of the same function is reduced. However, SAGE still could rely on the parallel setup method and multi-stage resource exit method to optimize the function latency. With these two methods, the function’s invocations could still benefit from reduced setup time and a warm context environment.

At the same time, experimental results also show that SAGE outperforms FixedGSL and DGSF in terms of system throughput by 9.7$\times$ and 1.19$\times$, respectively. This also comes from the possible read-only memory sharing and multi-stage resource exit methods. While the memory sharing opportunity is reduced, SAGE could still utilize it to improve the system throughput. In addition, the multi-stage resource exit mechanism could keep the resources warm for a longer time, which also helps to improve the system throughput.

To evaluate the effectiveness of the parallelized function setup mechanism, we measure the end-to-end duration of the GPU functions like the experiment in §3.2.1. The function invocations are generated in a close-loop manner.

Figure 15 shows the duration breakdown of all the benchmark functions with SAGE. Compared with the results in Figure 2, SAGE reduces the setup time of all the functions by 20.8% on average. This result could also be observed from the figure. As for the benchmark functions, they either have the GPU context preparation time or the data preparation time. This is because the parallelized function setup mechanism hides another setup stage, which improves the function setup performance. Benefiting from the parallelized function setup mechanism, SAGE could improve the 99%-ile latency of all the benchmark functions.

As shown in Table 4, the invocations could be in four stages or in a cold state. We use Resnet50 as the representative benchmark to show the breakdown in different stages. Compared with the traditional method which only includes warm state and cold state, SAGE contains five stages for fine-grained resource management. As shown in the table, the bold values and the zero values are hidden by the Italic value in the mechanism. For example, the GPU data loading time (67.2) and the CPU data loading time (21.7) are hidden by the GPU context creation time (285.1) in stage 4.

SAGE improves the function’s latency by an average of 6.1$\times$ in the other four states compared to the cold state (baseline), with a minimum of 1.3$\times$. All other benchmarks show similar results. The experimental results show that SAGE reduces 91.7% of the queries in the cold state. Therefore, the multi-stage resource exit mechanism could improve the function’s 99%-ile latency.

In this subsection, we disable the read-only memory sharing of SAGE (denoted as SAGE-NR), and compare its latency and system throughput with DGSF and SAGE. In this experiment, we extend DGSF to destroy the context while there are no function invocations in 30 seconds.

Figure 16 shows the average latency of 10 functions under DGSF, SAGE-NR, and SAGE. On average, SAGE outperforms SAGE-NR and DGSF by 8.2$\times$ and 13.3$\times$, respectively. Without the read-only memory sharing method, SAGE-NR still needs to suffer resource contention in the data loading paths. Meanwhile, although DGSF could eliminate the context creation time using pre-created context, it still suffers from context creation time when there is no created context. While SAGE-NR could utilize the multi-stage resource exit and parallelized setup preparation methods, SAGE-NR could perform better than DGSF. Furthermore, SAGE-NR has a similar memory usage with DGSF. SAGE-NR’s memory usage is only 38.3% of SAGE. s

To assess SAGE’s scalability, we conduct the performance tests on a cluster with 4 A100 devices on 4 nodes. Figure 17 illustrates the average latency of ten applications using FixedGSL, DGSF, and SAGE on the cluster. On average, SAGE surpasses FixedGSL and DGSF by 207.1$\times$ and 12.5$\times$, respectively. Experimental results also show that SAGE delivers a 10.3$\times$ and 1.18$\times$ increase in system throughput compared to FixedGSL, and DGSF, respectively.

Based on the aforementioned data, it can be deduced that SAGE has excellent scalability. Although we dispatch the function invocations to GPU randomly, SAGE could still improve the overall system’s average latency and system throughput. This is because the proposed optimization methods of SAGE are orthogonal to the task distribution strategy at the cluster level. SAGE can integrate any cluster-level scheduling strategies proposed for specific scenarios to further achieve performance improvements.

The overhead of SAGE arises from the runtime interception of functions and the communication between the kernel executor and memory daemon. Since the runtime interception of functions has been widely used in the past decade, its overhead is very small. At the same time, we use shared memory to support the communication between the executor and daemon. The communication overhead is microsecond-level. Therefore, the overhead of SAGE is negligible.

We have got some other lessons during the research.

Lesson-1: The fine-grained GPU memory management also enables the fine-grained pricing model for GPU serverless. While memory is crucial for GPUs, it is more reasonable to charge the users based on both the memory usage (the size of memory $\times$ hours) and computation usage. In this case, the pricing model of traditional FixedGSL is not reasonable as a GPU function may not need 1GB memory.

Lesson-2: While a GPU often has many streaming multiprocessors (SMs), SAGE can be extended to explicitly allocate SM allocation to GPU functions or adjust the processing order of GPU serverless functions. The functionality can be done through the shim module and provides better control over the progress of concurrent GPU functions.

Lesson-3: While SAGE’s fine-grained memory management enables GPU to have more memory available, it could be extended to use a cache replacement strategy for container warm-up. For a new-coming function, we can use a cache replacement strategy to replace long-inactive GPU functions. Meanwhile, while there are no new GPU functions arriving, existing GPU functions could always reside in GPU memory.