Container Profiler: Profiling Resource Utilization of Containerized Big Data Pipelines

Large-scale and diverse biomedical data have been generated to advance the understanding of biological mechanisms. Interpreting these data typically includes multiple analytical steps, each of which consists of different computational methods and software tools. An analytical pipeline (or workflow) is a sequence of computational tasks used to process and analyze specific biomedical data. Each analytical step in a pipeline can potentially require a different set of applications, libraries, and software dependencies. As a result, software containers that encapsulate executables with their dependencies have become popular to facilitate the deployment of complicated pipelines and to enhance their reproducibility  [29, 38]. Additionally, different analytical steps in a pipeline could have different computing resource requirements. In particular, many bioinformatics pipelines consist of one or more computationally intensive steps stemming from their operation on large datasets requiring significant CPU, memory, network, and disk resources. As an example, the alignment step in a RNA sequencing pipeline typically requires relatively more CPU and memory resources than other steps, while the data download step typically requires more disk I/O and network resources.

Cloud computing has emerged as a solution that can provide the necessary resources needed for computationally intensive bioinformatics analyses  [13, 34, 33, 23, 31, 9, 10]. However, deployment of analytical pipelines using Infrastructure-as-a-Service (IaaS) cloud platforms requires selecting the appropriate type and quantity of virtual machines (VMs) to address performance goals while balancing hosting costs. Cloud resource type selection is presently complicated by the rapidly growing number of available VM instance types and pricing models offered by public cloud providers. For example, the Amazon, Microsoft, and Google public clouds presently offer hundreds of different VM types under different pricing models. Further, Google allows users to create custom VM types with unique combinations of CPUs, memory, and disk capacity. These cloud VMs are available directly, or through various container platforms. Determining the best cloud deployment requires understanding the resource requirements of the pipeline.

This paper presents the Container Profiler, a tool that supports profiling the computational resources utilized by software within a Docker container. Our tool is simple, easy-to-use, and can record the resource utilization for any Dockerized computational job. Understanding fine-grained resource utilization of containerized computational tasks can help identify resource bottlenecks and inform the choice of optimal cloud deployment. The Container Profiler collects metrics to characterize the CPU, memory, disk, and network resource utilization at the VM, container, and process level. In addition, the Container Profiler supports time-series graphing enabling the visualization and monitoring of resource utilization of containerized tasks and pipelines. We present a case study using a multi-stage containerized bioinformatics pipeline that analyzes the unique molecular identifiers (UMI) of RNA sequencing data to illustrate the utility of our tools.

Key Points

• •

  We present the Container Profiler a tool that enables profiling the resource utilization of any script or container-based task on Linux.

• •

  The Container Profiler collects CPU, memory, disk, and network resource utilization metrics at the virtual machine, container, and process levels.

• •

  The Container Profiler supports delta and time series resource utilization profiling at an adjustable time interval (e.g. 1-second) supporting monitoring and graphing of resource utilization enabling time series analysis to help identify performance bottlenecks for any Linux-based computational task.

• •

  The Container Profiler can profile complex containerized computational jobs such as bioinformatics pipelines where multiple individual containers are used to implement specific steps.

• •

  The Container Profiler is provided as a container which can merge with any existing container or used separately to profile independent Linux scripts or executables to characterize task resource utilization locally or on the cloud.

• •

  We illustrate how different resources are required when performing different steps of a containerized pipeline analyzing unique molecular identifiers (UMI) RNA sequencing data.

Figure 2 summarizes the CPU utilization characteristics of different stages of the UMI RNA-seq pipeline. The CPU usage profile is consistent with our expectations. The execution of the align and merge steps are expected to be bound by CPU resources and they indeed spent the majority of the time executing source code. Download is limited by the network bandwidth and the split stage by disk I/O. Hence the cpuIdle time is highest in these stages.

Despite the fact that the align stage is expected to be limited by the CPU resources, there is significant CPU-idle time during that stage. This suggests the presence of a bottleneck that may be the target for further optimization. We collected CPU, memory, network, and disk utilization metrics at both the container and VM/host levels for the RNA sequencing analytical pipeline. These are visualized in Figure 3. Note that the x-axis depicting time in this figure encompasses the entire pipeline incorporating all stages: download, split, align, and merge. Overall our profiling results depict resource utilization patterns that we expected. The download stage consumes network resources. The split stage is the most disk intensive step. The alignment and merge stages consume the most CPU resources. Our profiling data also points to areas where resource consumption may be a problem. For example, memory usage is high for all the stages. This may be due to greedy allocation by the executables, or it may indicate that allocating more memory could benefit the pipeline. Most interesting, is CPU utilization during the alignment stage. Just before the 3 hour mark, we see a series of drops over the next 30 minutes, creating a ladder of 8 steps. The alignment stage uses up to 8 vCPUs to align different files of reads simultaneously. Near the end of the alignment stage, most of the files will have been processed and there will be more available vCPUs than unprocessed files. As a result, the CPU utilization drops as vCPUs lie idle waiting for the final files to be processed. However, this under-utilization of resources lasts for 30 minutes indicating that the final files are rather large. This presents an opportunity to improve pipeline performance by splitting the processing into smaller files (which is an option in the split software), or by processing the largest files first. We would not have known about these potential optimizations without fine-grained profiling results from the Container Profiler.

A key feature of the Container Profiler is the ability to capture container-level metrics to describe resource utilization of only the containerized task(s). We expect these metrics to be similar, and that they could differ given that the VM/host level metrics also encompass resources being used by processes running on the host external to the container and pipeline. Since we only ran our pipeline on an dedicated test VM, the container metrics should be very similar to the VM/host metrics, which was in fact the case from our observations. However, one can see differences between the disk utilization metrics during the split and alignment stages where there are a large number of disk writes to the host file system. Docker manages these disk writes by providing the container with an internal mount point which is eventually written to a host file. The caching and management of this data is external to the container and is not captured by the container metrics, but is captured by the host metric. In addition, during the alignment stage, intermediate results from the aligner are continuously piped to another process which then re-formats the intermediate output and writes the final output to a file on the host system. Multiple threads are used, more than the available number of cores resulting in frequent context switches. The pipe management and context-switching are also handled by the operating system and are captured by the host metric and not the container metrics. The separation of container and OS based consumption can be useful for example, when trying to assess effects due to resource contention that may occur when multiple jobs are run on the same physical host, which often happens on public clouds where the assignment of instances to hosts is controlled by the vendor.

For the Container Profiler to be useful, the collection of profiling metrics must have sufficiently low overhead to enable rapid sampling of resource utilization to collect many samples for time series analysis. The time required to collect the metrics limits the granularity of the profile. To achieve 1 second sampling for time series analysis requires the ability to repeatedly sample resource utilization every 1 second (1000 ms). However, profiling time is not constant, and depends on the state of resources being utilized by the containerized pipeline and the host. The variability of profiling time is shown in the histogram in Figure  4. When profiling our RNA-sequencing pipeline, VM-level and container-level profiling had a bi-modal distribution, while process-level sampling had a tri-modal distribution. The slowest profiling was observed during the stressful compute-bound alignment stage of the pipeline.

For all levels of profiling verbosity, the Container Profiler was able to profile resource utilization in less than 100ms. The longest profiling time and highest variation was for process-level profiling as metrics are collected for each process in the pipeline. The number of processes can vary throughout the execution of complex parallel pipelines, as was the case for the align stage of our RNA-sequencing pipeline. Our RNA-sequencing pipeline featured a maximum of 85 concurrent processes during the align stage. These processes ran for approximately 39% of the duration of the align stage. The time required to capture host and container level metrics was less variable as the number of metrics collected is fixed. As shown in Figure  4, 90% of the time, the container and host level metrics were collected in less than  63 milliseconds and always under 75 milliseconds. The process metrics do take longer to collect but still less than 100 milliseconds in the worst case. Profiling at the process-level involves collecting all metrics every second. For profiling our UMI RNA-sequencing pipeline use case which required  2.5 hours to execute with one-second sampling and full profiling verbosity (process-level metrics),  9,000 JSON files were collected which required 296 MB of storage space.

A design objective for the Container Profiler is to not significantly impact the performance of the pipeline being profiled. Failing to realize this objective may result in the overhead from resource profiling impacting the collected metrics. While some overhead is unavoidable, ideally it should be lower than the inherent variations of pipeline execution time on the public cloud.

To measure the performance impact of resource utilization profiling when running the RNA-seq pipeline, we initially attempted to assess the overhead using Amazon Elastic Compute Cloud (EC2) cloud VMs. However, we discovered that the runtime of the RNA-seq pipeline varied by more than 5% on Amazon EC2, which was more than 5x greater than the overhead of the Container Profiler. This degree of performance variance made it difficult to evaluate the performance overhead of the Container Profiler since we could not easily distinguish between pipeline performance variance and profiling overhead on EC2. We then measured the performance overhead of the Container Profiler by profiling the pipeline using the IBM cloud bx2d-metal-96x384 server which had performance variance around  1%. Figure 5 depicts the overhead from one-second resource utilization sampling by the Container Profiler for the RNA-seq pipeline on the IBM metal server. IBM metal servers are private and not shared with multiple users. Running on this isolated server greatly reduced the performance variance of running RNA-seq. We measured worst case overhead for the Container Profiler to be 0.71%, which equates to about 3.4 minutes for an 8-hour pipeline with full verbosity metrics collection (VM + container + process). Overhead is reduced to as little as .07% overhead, or about 20 seconds for an 8-hour pipeline when only collecting VM-level metrics. Adding container-level, and especially process-level metrics slightly increased the runtime of the RNA-seq pipeline for collecting resource utilization data. We believe that this profiling overhead is within an acceptable level and note that even at maximum profiling verbosity, it is substantially less than the observed performance variance for running our RNA-seq pipeline on a public cloud VM. Users can reflect on our reported overhead times to make informed decisions when planning to profile their own pipelines.

The Container Profiler is implemented as a collection of Bash and Python scripts. Figure 6 provides an overview. There are three basic use cases for building a docker image for the Container Profiler. The first use case allows users to profile an existing Docker container by providing a Dockerfile which specifies their own setup and software installation inside the container. This is the simplest approach to profiling when the user has a working Dockerfile. The other two use cases support users who do not know how to write Docker files but are familiar with writing Bash scripts. The second use case gives users the ability to install all software inside the Docker image when the software installation becomes too complicated to put in the Dockerfile. In other words, it puts the required installation commands into a script that will be executed by the Dockerfile. For the third use case, the user provides their own executable bash script as the entry point in the Docker container. This use case can help the user simplify a set of commands they have to profile. In this case, the user just puts a set of commands into an executable script file and runs it as the entrypoint of the container. When the Container Profiler is executed inside a Docker container, it snapshots the resource utilization for the host (i.e. VM), container, and all processes running inside the container producing output statistics to a .json file. A sampling interval (e.g. once per second) is specified to configure how often resource utilization data is collected to support time series analysis of containerized applications and pipelines. Time series data can be used to train mathematical models to predict the runtime or resource requirements of applications and pipelines. Time series data can be visualized by using matplotlib Python graphing scripts that are included with the Container Profiler.

To improve the periodicity of time series sampling, we continuously subtract the most recent observed run time of the Container Profiler for sample collection from the configured sampling interval (e.g. 1 second) in rudataall.py. This approach notably improved the periodicity of sampling when the container was under load improving our ability to obtain samples at evenly spaced intervals. To enable addressing any potential drift of sample collection times, we capture timestamps for when each resource utilization metric is sampled in the output JSON. These timer ticks enable precise calculation of the time that transpires between resource utilization samples for each metric. This allows the rate of consumption of system resources (e.g. CPU, memory, disk/network I/O) to be precisely determined throughout the pipeline’s execution. The Container Profiler consists of the profiling script and two supporting scripts (for installation and pipeline execution) depicted in Figure 6: profiler.sh, install.sh, and execute.sh.

The profiler.sh script is the primary script that generates profiling information in JSON format describing resource utilization of the containerized task. The profiler.sh script requires the user to provide a command or a set of commands along with arguments to start the profiling. This script internally invokes another Python script rudataall.py.

The rudataall.py script collects the resource utilization data. Specifically, this script takes a snapshot of the resource utilization metrics and records output to a JSON file using the time of the sample as a unique filename. The script accepts parameters -v, -c, and -p to inform the tool what type of data to collect: VM, container, and/or processlevel metrics respectively. The default behavior when running this script without any parameters is to collect all metrics.

The profiler.sh script only works if the workflow/software is already installed in the containerized environment. This means that we cannot profile workflows/software that has not been containerized. The Container Profiler provides an option that enables users to install software in a container using the install.sh script. Users provide a set of commands in the install.sh script to install their dependencies and software they wish to profile. Once installed, the user can run the profiler.sh script against the newly installed software. To profile resource utilization of a bash script, users can specify a series of commands using the optional execute.sh script to configure profiling.

Some users may be more familiar with editing Dockerfiles instead of bash scripts. We provide support for users to provide their own Dockerfile to build a custom container to be profiled.

To use the Container Profiler scripts with any container, a Linux based Docker container that encapsulates a script or job to run inside is required. To configure the Container Profiler tool to profile the container, users can optionally provide an executable script inside the Container Profiler which is specified during the build.sh script. In the executable script, the user launches the container’s job or task to be profiled.

The profiler.sh script has four different modes: profile, delta, csv, and graph:

For the profile mode, there are two required parameters: the output directory specifies the location of generated profiling files in JSON format, and the time interval specifies a time series sampling interval in milliseconds. The profiler generates a JSON file at the beginning and the end of the process if the sampling interval is set to zero. Otherwise, the profiler generates a JSON file at each sampling interval. The Container Profile also collects static metrics which typically describe hardware characteristics. The profiler first checks if a static information file exists (static.json). If missing, the profiler captures static parameters and writes out the static information file at the start of profiling. By default 11 static metrics are captured. They include: the host’s kernel info, the host’s cpu type, CPU Level 1 instruction cache size, CPU Level 1 data cache size, CPU Level 2 cache size, CPU Level 3 cache size, host boot time, host VM ID, the number of CPU cores available to the container, and the container ID.

For the delta mode, there are two required parameters: the input directory which contains the original raw JSON files, and the output directory where the delta JSON files will be written. The delta| mode also provides an option to allow users to specify the modification operator for performing the delta. The default delta operator calculates the difference between two samples (i.e. final minus initial value). The typical use case is to calculate the delta of the resource utilization between the first and last sample to capture the full resource utilization of a task or pipeline. Other operators include max, min, and average to determine the max, min, and average values of metrics from a set of JSON files.

For the csv mode, there are two required parameters: the input directory that contains processed JSON files in delta format, and the name of an output CSV file where all resource utilization data from the processed JSON files will be aggregated to.

For the graph mode, there are two required parameters: the input CSV file capturing all resource utilization data from processed JSON files, and the output directory for writing graph files. In addition, there are a few other options such as one to specify whether to plot the curves together or using separate graph files.

The Container Profiler in the graph mode also provides an option to specify the creation of time-series graphs. The graphing configuration file supports multiple settings to specify how to generate graph(s). Each graph configuration file should start with a line that includes the components: the ### followed by the title and the y-coordinate label. This is followed by line(s) that describe the metric(s) that users want to output in a single graph (one metric per line). As a starting point, a default graph configuration file graph.cfg is provided in the cfg directory.