On-demand Cold Start Frequency Reduction with Off-Policy Reinforcement Learning in Serverless Computing

In cloud computing, a serverless deployment model removes the burden of managing and provisioning resources from the developers, allowing them to focus solely on the application development process. The term serverless, interchangeably used with Function-as-a-Service (FaaS), does not imply an absence of servers, but instead, accentuates delegating the responsibility of complex resource management tasks to cloud service providers (CSP) [16], [13]. The FaaS paradigm puts forward an event-driven, serverless computing model with fine-grained pay-per-use pricing where resources are billed based on their actual service time. Functions (i.e., a fragment of code containing business logic) are designed to scale on demand; they are stateless, short-lived, and run on lightweight containers or virtual machines (VMs) in response to a triggering event. Such an abstraction increases agility in application development, offering lower administrative and ownership costs. The FaaS model has attracted a wide range of applications such as IoT services, REST APIs, stream processing and prediction services, which have strict availability and quality of service requirements in terms of response time. Conceptually, the FaaS model is designed to spin a new function instance for each demand request and shut down the instance after service [13]. However, in practice, commercial FaaS offerings like AWS Lambda, Azure Functions, and Google Cloud Function may choose to re-use a function instance or keep the instance running for a limited time to serve subsequent requests [33]. Some open source serverless frameworks like Kubeless [1] and Knative, have similar implementations to re-use an instance of a function to serve subsequent requests.

An increase in workload demand leads to an instantiation process involving the creation of new function containers and the initialisation of the function’s environment within those containers, after which incoming requests are served. Such a process usually requires downloading the client code, setting up code dependencies and the runtime environment, setting up container networking, and finally executing the code to handle the incoming request. Hence, instantiating a function’s container introduces a non-negligible time latency, known as cold start, and gives rise to a challenge for serverless platforms [36], [15], [10], [27]. Some application-specific factors such as programming language, runtime environment and code deployment size as well as function requirements like CPU and memory, affect the cold start of a function [27], [22], [20], [31]. To automate the process of creating new function instances and reusing existing ones, serverless frameworks usually rely on resource-based (CPU or memory) horizontal scaling, known as horizontal pod auto-scaling (HPA) in Kubernetes-based frameworks like Kubeless, to respond to incoming requests. Resource-based scaling policies implement a reactive approach and instantiate new functions only when resource usage rises above a pre-defined threshold, thus leading to cold start latencies and an increase in the number of unsuccessful requests.

Threshold-based scaling decisions fail to consider factors like varying application load and platform throughput and hence, pose an opportunity to explore dynamic techniques that analyse these factors to address cold starts. This work presents a model-free Q-learning agent to exploit resource utilization, available function instances, and platform response failure rate to reduce the number of cold starts for CPU-intensive serverless functions. We define a reward function for the RL agent to dynamically establish the required number of function instances for a given workload demand based on expected average CPU utilisation and response failure rate. The RL-based agent interacts with the serverless environment by performing scaling actions and learns through trial and error during multiple iterations. The agent receives delayed feedback, either positive or negative, based upon the observed state, and consequently learns the appropriate number of function instances to fit the workload demand. This strategy uses no prior knowledge about the environment, demand pattern or workload, and dynamically adjusts to the changes for preparing required functions in advance to reduce cold starts. The proposed work scales the number of function instances by proactively estimating the number of functions that are needed to serve incoming workload to reduce the frequent cold starts. It utilizes a practical workload of matrix multiplication involved in an image processing task, serving as a sample real-world function request pattern [28], and formally presents the cold start as an optimisation problem. Also, we structure the Q-learning components around the function metrics such as average CPU utilisation and response failure rate and evaluate our approach against the default resource-based policy and commercially accepted function keep-alive technique.

In summary, the key contributions of our work are:

1. 1.

   We analyze function resource metrics such as CPU utilization, available instances, and the proportion of unsuccessful responses to propose a Q-learning model that dynamically analyses the application request pattern and improves function throughput by reducing frequent cold starts on the platform.

2. 2.

   We present a brief overview of explored solutions to address function cold starts and highlight the differences between contrasting approaches to the proposed agent.

3. 3.

   We perform our experiments on a real-world system setup and evaluate the proposed RL-based agent against the default resource-based policy and a baseline keep-alive technique.

The rest of the paper is organised as follows. Section 2 highlights related research studies. In Section 3, we present the system model and formulate the problem statement. Section 4 outlines the proposed agent’s workflow and describes the implementation hypothesis and assumptions. In Section 5, we evaluate our technique with the baseline approach and highlight training results and discuss about performance in Section 6. Section 7 highlights future research direction and Section 8 summarises and concludes the paper.

In this section, we briefly discuss about the Function-as-a-Service paradigm in serverless computing and elaborate on the current function cold start mitigation techniques and approaches.

FaaS is an event-driven cloud service model that allows stateless function deployment. It delivers high scalability and scale-to-zero feature being economical to infrequent demand patterns. New functions $n_{i}$, where $1\leq n_{i}\leq N$ and $N$ is the maximum scale, are instantiated on-demand to serve the incoming load (scale up) and removed (scale down) when not in use after a certain time span or below a configured, resource-based threshold metric value for every $i$ iteration window. The preparation time of function containers i.e., cold start $C_{t}$, adds to the execution time of a request. These frequent on-demand cold starts result in an increased computation pressure on existing resources, neglecting expected average CPU utilisation ($\phi_{o}$), and expected request failure rate ($\tau_{o}$). Therefore, an intelligent, learning-based solution is proposed to address them.

In this study, we consider Kubeless, an open-source Kubernetes-native serverless platform that leverages Kubernetes primitives to provide serverless abstraction. It wraps function code inside a docker container with pre-defined resource requirements i.e. $RR_{f}=(cpu_{f},mem_{f},tout_{f})$ and schedules them on worker nodes. Similar to commercial FaaS providers, Kubeless has an idle-container window of 5 minutes to re-use functions and scales down to a minimum of one function if the collected metrics (default 15 seconds window) are below the set threshold. We take into account the general illustration of FaaS platform and consider a stochastic incoming request pattern $D=\{d_{1},d_{2},\dots,d_{i}\}$ with $d_{i}$ requests in $i^{th}$ iteration window. We analyze the request pattern for a timeframe $T$ divided in $i$ iteration windows of duration $t_{i}$. The system model of the examined scenario is depicted in Fig. 1. The workflow of the potential cold start is explained in Fig. 2.

In this work, we apply model-free Q-learning algorithm in FaaS paradigm to reduce frequent on-demand function cold starts. We select this algorithm due to its simple and easy implementation, model interpretability, strong theoretical convergence guarantees, ability to process the perceived information quickly using the Bellman equation and its adaptability to other advanced algorithms like Deep Q-Learning (DQN). As discussed in Section 3, we model the process of creating required function instances as an MDP and map the serverless computing primitives to RL agent’s environment, state and actions. We explore and exploit an off-policy RL algorithm to reduce the on-demand function cold starts and determine the required function instances with the intuition of it being easy, simple to implement, less complex with stable convergence in a discrete action space. The proposed approach has two phases: an agent training phase and a testing phase. Algorithm 1 demonstrates the agent training workflow. The environment setup process precedes the agent training, where the agent interacts with the environment and obtains information. After initial setup, the agent is trained for multiple epochs or timeframes where it assesses the function demand $d_{i}$ over individual iteration windows $i$ and ascertains appropriate function instances. During an iteration window $i$, the agent observes the environment state $s_{i}$, selects an action $a_{i}$ according to $\epsilon$-greedy policy. This greedy policy helps the agent to control its exploration and selects a random action with $\epsilon$ probability, otherwise exploiting the obtained information. This exploration rate is a dynamic value and decays with ongoing learning to prioritise the acquired information.

After performing the selected action, the agent waits for duration $t_{i}$ of an iteration window to obtain the delayed reward $r_{i}$, calculated using the relevant resource-based metrics $\phi_{d_{i}}$ and function failure rate $\tau_{d_{i}}$. This reward helps the agent in action quality assessment, and it combines the acquired knowledge over previous iterations using the Bellman Equation (Eq. 5). It is the core component in learning as it aids Q-value or Q-table updates and improves the agent’s value-based decision-making capability. The equation uses two hyper-parameters learning rate, $\alpha$ and discount factor, $\gamma$. The learning rate signifies the speed of learning and accumulating new information, and the discount factor balances the importance of immediate and future rewards.

The agent then evaluates and adjusts the Q-value in Q-Table based upon the delayed reward for the corresponding ($s_{i},a_{i}$) pair. The agent continues to analyse the demand over multiple iteration windows, selecting and performing actions, evaluating delayed rewards, assessing the quality of action and accumulating the information in Q-table, and repeating this process over multiple epochs for learning. Once the agent is trained for sufficient epochs and the exploration rate has decayed significantly, we can exploit obtained knowledge in the testing phase.

In the testing phase, the agent is evaluated using a demand pattern for the matrix multiplication function and the Q-table values guide the agent in taking informed actions. The agent determines the current environment state and obtains the best possible action i.e. action with the highest Q-value for the corresponding state, and adjusts the required number of functions based on its understanding of the demand. We hypothesise that there exists a relationship between throughput and function availability to serve incoming requests. Therefore, we evaluate the agent’s performance by considering metrics such as system throughput, function resource utilisation and available function instances. We further hypothesise that the RL-based agent learns to prepare and adjust required number of functions beforehand and improve the throughput while keeping the function’s resource utilisation below the expected threshold.

In this section, we provide the experimental setup and parameters, and perform an analysis of our agent compared to other complementary solutions.

Function cold start is an inherent shortcoming of the serverless execution model. Thus, we have proposed an RL-based technique to investigate the demand pattern of the application and attempt to reduce the frequency of function cold starts. The proposed agent performs better than the baseline approaches under a controlled experimental environment. But there are certain points to recollect associated with the real-time appropriateness of the proposed solution.

We leverage the RL environment modelling, specifically Q-Learning constraints [32], [37], and in general, these algorithms are expensive in terms of data and time. The agent interacts with the modelled environment to acquire relevant information over multiple epochs that signify a higher degree of exploration. Hence, as evidenced in the proposed work, for an RL-based agent to outperform a baseline technique, a training period of 500 epochs is exploited for satisfactorily analyzing the workload demand for a timeframe (10 minutes). Therefore, RL-based approaches are considerably expensive in practical applications with stringent optimization requirements.

A classical Q-Learning approach is applied to discrete environment variables [32]. To constrain the serverless environment within the requirements of the Q-Learning algorithm, we consider the discrete variables to model cold starts. The size of the Q-table is large and is a function of state space and action space. But with the expansion of state space or the action space, the size of the Q-table grows exponentially [32], [37]. Therefore, Q-Learning experiences state explosion, making it infeasible to perform updates on Q-values and degraded space and time complexity.

The proposed agent analyzes individual application demand, so the learning can’t be generalized for other demand patterns and requires respective training to be commissioned. Furthermore, the agent is trained for 500 iterations and evaluated, but the chance of exploring every state is bleak with limited iterations of training. Therefore, the agent expects to be guided by certain approximations to avoid acting randomly. The agent utilizes resource-based metrics that affect the cold starts, so the availability of relevant tools and techniques to collect instantaneous metrics is essential. Also, the respective platform implementation of a serverless environment, such as metrics collection frequency, function concurrency policy, and request queuing, can extend support to the analyses.

The difference between the approaches can be attributed to the following characteristics of the proposed RL-based agent –

1. 1.

   The process of elimination of invalid states during the RL environment setup and lazy loading of Python, helps the agent to productively use the acquired information about the environment.

2. 2.

   Although the RL-based agent outperforms HPA and function queue policy, there is a lack of function container concurrency policy. The CPU-intensive function workload is configured with an execution time of 60 seconds and thus affected by the concurrency control of the instance.

3. 3.

   The composition of state space and reward function incorporates the effect of failures during the training, and therefore, the agent tries to compensate for the failures in consequent steps of learning by exploiting the acquired knowledge.

On the account of the performance evaluation results, we can adequately conclude that the proposed agent successfully outperforms competing policies for the given workload and experiment settings. We strengthen this claim by analysing the training and testing outcomes of the RL-based agent, focused on examining the workload pattern to reduce request failure which is a direct consequence of appropriate function instances representing reduced function cold starts.

As part of the future research in broader serverless computing domain, various aspects of resource management needs to be addressed. These could be broadly categorised under workload estimation and characterisation, resource scheduling, and resource scaling. Following the increased adoption of FaaS, challenges such as function cold start delay, co-located function interference, lack of QoS guarantees, user pricing model, runtime limitations and support for specialised hardware, resource efficiency and workload management emerge as concerning factors for the success of serverless computing [19]. With the changing dynamics of application workload, its resource demand and introduction of AI/ML models in real-time systems, there is need for adaptive and proactive methods to tackle inefficient resource management in serverless computing. Additionally, there is a need for autonomous resource management system to completely offload the function scheduling, resource allocation and resource scaling tasks to offer a true serverless experience for developers and utilise the provider resources to the maximum. However, a guarantee of QoS objectives is still warranted in terms of service latency, throughput, fault tolerance and cost incurred by the user. Other aspects of FaaS that require meticulous thinking with adoption in edge and fog computing environments are data privacy and data locality concerns that further reduces the flexibility of FaaS application model.

Specifically, to address the issue of function cold start delays, an impact of other important aspects such as function memory allocation, language runtime, deployment size, programming convention and function characteristics in conjunction with different techniques can be explored as discussed below:

• •

  In the industrial works like [3], [4], an insight is provided for the serverless AWS Lambda functions and how an improvement can be made to its cold start latencies and operating costs with careful function initialisation phase consideration.

• •

  An in-depth analysis of function fusion or monolithic function development on resource requirements and cold start of function workflows can also be explored with respect to the performance and run-time costs of the function.

• •

  Furthermore, a trade-off analysis of techniques like function pre-baking, provisioned concurrency and reserved concurrency can also be valuable in performance optimisation of serverless function and associated application workflows.

• •

  In addition to CPU-intensive functions, impact of cold start on different classes of functions like memory-intensive, I/O-intensive and network-intensive functions, utilised for application domains such as AI/ML model training and inference, and media processing, can also be explored.

• •

  Consequently, a cost and performance analysis of RL-based agents can also be made for the function cold start reduction in these application cases. Similar to Q-Learning, the application of other policy-based techniques such as SARSA, which is known to converge faster than Q-Learning, can also be experimented with in the domain of the cold start problem. As an adaptation of Q-Learning, the proposed solution includes discrete values over continuous values for state representation. In this context, to avoid the problem of state space explosion, function approximation techniques such as DQNs, Proximal Policy Optimization (PPO) and other complex deep learning methods such as Soft-Actor-Critic (SAC) or Recurrent models [6] can also be leveraged to estimate the information about optimal actions.

• •

  To further explore the evolving technological space, an integration of Generative Adversarial Networks for synthetic training data generation or Large-Language-Models can be explored for pro-active cold start reduction using the historical data.

FaaS model executes the piece of code inside a container, known as a function and prepares new function containers on demand. New function containers undergo an initialisation process that puts together all the essential components before executing the function handler. This bootstrapping process consumes time in the order of a few seconds, known as function cold start and introduces a delay in the response of the function container.

This work visits the problem of function cold start by addressing frequent cold starts and analysing the application demand through an RL technique. We leverage the services of Apache JMeter to produce varying incoming request patterns and a CPU-intensive function workload to complement the invocation pattern and observe relevant cold starts. The system is set up using the Kubeless framework and the RL environment is modelled for the agent to examine the necessary metrics to make guided decisions in provisioning an appropriate number of function instances.

We present an evidence of leveraging Q-Learning to address cold starts on FaaS platforms and verify it with improved platform throughput, reduced resource wastage while maintaining expected thresholds, during the iterations. We evaluate the performance of our proposed agent against the HPA policy and function queue policy. We successfully observe the RL-based agent outperforming comparing techniques after a training of 500 epochs that verifies our hypothesis of strong association between success rate and reduced number of cold starts on the platform. After the test analyses, the Q-Learning agent successfully improves throughput by up to 8.81% and reduces resource wastage by up to 37% while preparing sufficient functions to reduce cold starts.