ML-based Modeling to Predict I/O Performance on Different Storage Sub-systems

Burst Buffers (BBs) are fast intermediate storage layers between compute nodes and storage nodes [3]. The traditional HPC I/O architecture consists of on-node memory, I/O nodes for handling I/O requests, and a disk-based parallel file system [10]. But this architecture is not enough to fill the gap between computational performance and I/O performance [7]. Therefore, BB is a natural solution to be introduced between these layers. BBs are implemented on several state-of-the-art HPC clusters such as Lassen, Summit, and Cori.

Burst buffers can be implemented in various ways. In terms of hardware, BBs are mostly made with SSDs. For example, Lassen at LLNL uses 1.6 TB Samsung PM1725a NVMe SSDs [11]. In terms of architecture, there are two major types: node-local and shared BBs [12]. Lassen and Summit adapt node-local BBs. In this case, a BB is directly connected to a compute node. All BBs are independent, which means they don’t share the same namespace. For the shared BBs (as used in Cori), BB nodes are separated from compute nodes and can be accessed by multiple compute nodes through interconnected networks [13]. In this case, compute nodes perform I/O on the global BB file system with a shared namespace. In this paper, we focus on node-local BBs.

Research has demonstrated the benefit BB brings to I/O performance [2]. For example, Bhimji et al. analyze the performance of a variety of HPC applications on Cori burst buffers, including scientific simulations, visualization, image reconstruction, and data analysis [7]. They demonstrate that burst buffers can accelerate the I/O of these applications and outperform regular Parallel File Systems (PFS). Pottier et al. [14] also demonstrate that node-local burst buffer significantly decreases the runtime and increases the bandwidth of an application called SWarp. From the research mentioned above [2, 14, 7], it is evident that BB has the potential to improve the I/O performance of certain HPC applications.

Although these papers demonstrate BB is promising in improving I/O performance, most of them raise the same concern: The performance is sensitive to application I/O patterns [7, 8, 14]. Pottier et al. demonstrated different workflows may not get the same benefit from BBs [14]. Some of them have decreased performance compared with the performance on regular file systems. For instance, they indicated that the shared burst buffer can perform worse than the regular PFS and is sensitive to the amount of data transferred per read/write operation. However, their work is only based on a single application called SWarp so it might not be general enough to cover common I/O patterns of HPC application. Existing research on this issue has not given a general solution to the problem due to insufficient experiments and modeling to cover different I/O patterns. Besides, the metric they use is the total runtime, which may not be a good I/O metric as it can be significantly affected by work other than I/O like computation. To summarize, the underlying relationship between an application’s I/O characteristics and the storage subsystem choice is still unclear.

There exist I/O performance analysis tools that can assist in studying such relationships. Two popular I/O performance analysis tools are PyDarshan [15, 16] and Recorder-viz [17]. Both of them trace application runs and capture basic performance information such as time and I/O volume of an operation. They provide visualization scripts upon their traces. With them users can derive aggregated I/O metrics such as I/O bandwidth, file sizes, etc. These tools can be helpful to start with the I/O subsystems selection research question, however, they are not efficient enough for detailed analysis due to the lack of interfaces that enable users to customize their analysis. Users have to make significant efforts to write their own codes to achieve deeper discoveries. Besides these two tracing tools, there is an automated analysis project called Tokio [18, 19], short for total knowledge of I/O. The project collects the I/O activities of all applications running on NERSC Cori for a time period and then provides a clear view of the system behavior. It does a great job in holistic I/O performance analyses such as detecting I/O contention and unhealthy storage devices at the system level. However, there is no application-level analysis so it does not map I/O characteristics to I/O subsystem selections.

We conduct our experiment on Lassen. Lassen is a cluster at LLNL using IBM Power9 CPUs with 23 petaflops peak performance. It uses IBM’s Spectrum Scale parallel file systems, formerly known as GPFS. GPFS is one of the most common implementations of parallel file systems (PFS). It is used by many of the world’s supercomputers [20]. In terms of BB, Lassen uses node-local BBs implemented with Samsung NVMe PCIe SSDs.

The benchmark we use is IOR [9]. We choose IOR because it is configurable enough to emulate various HPC workflows. It is proved to be an effective replacement for full-application I/O benchmarks [21] [22]. In our experiment, we select representative I/O characteristics that are common in real applications and configure IOR with them. For instance, we include collective MPI-IO because it’s a common characteristic in checkpoint/restart applications. We use IOR to emulate the I/O characteristics of real applications by covering all possible combinations of the selected I/O characteristics. We further separate I/O into six types, RAR, RAW, WAR, WAW, RO, and WO, to better map IOR runs with different types of applications (RAR for read after read, RAW for read after write, RO for read only, etc.). For instance, RAR and RO are common in ML applications between epochs. We again cover all feasible combinations of these read/write types and I/O characteristics.

We do the following to take variabilities into account. We repeat each run five times at different times and days to minimize the impact of temporal system variability. Since our whole experiment contains around 35,000 runs, we avoid adding abnormal burdens to the system and reflect system performance as normal as possible. We run them sequentially so at one time there is only one IOR run. We collect all I/O metrics from IOR standard output, including total I/O time, read/write time, bandwidth, etc, and take the mean of five iterations. Each IOR run produces a sample of the dataset.

The data we use is derived from IOR runs with different configurations on BB and GPFS, where each IOR run is a sample in the dataset. The independent variables are all I/O features we included in the experiment. Descriptions of them are presented in Table I. We first process them to make them suitable for common ML models. Most ML models cannot directly handle string-formatted categorical columns. Also, if a column is nominal instead of ordinal, simply converting it into integers ranging from 0 to the number of categories may result in erroneous inference. One popular solution is using one-hot vector encoding. It works by creating as many columns as the number of categories for a categorical column in the data set. Each column represents a category. We set a column to 1 if the original value is the category represented by that column and 0 for all other columns. We encode all our categorical columns such as file system, API, etc., into one-hot vectors.

As our goal is to classify the best I/O subsystem to place a file, the dependent variable should be the I/O subsystem. Recall that we run every I/O feature combination on both GPFS and BBs. We obtain the true label by selecting the I/O subsystem that gives the best I/O bandwidth for each particular I/O feature combination, which will be either GPFS or BB. The exhaustive I/O feature combinations plus the true label makes our final dataset. The final dataset has 19 feature columns and 2967 rows.

PrismIO is primarily designed for analyzing Recorder traces. Recorder [17] is an I/O tracing tool. It captures function calls from common I/O-related libraries such as fwrite, MPI_file_write, etc. We choose Recorder over Darshan because trace enables more detailed analysis than profiles. Recorder has a couple of APIs to generate plots, but it’s hard for users to efficiently customize analysis. To address this, we implement PrismIO as a Python library that builds on top of Pandas dataframe to organize data and build APIs upon it. The primary class that holds data and provides APIs is called IOFrame. It reorganizes the trace into a Pandas dataframe. Each row is a record of a function call. Columns store information of function calls, including start time, end time, the rank that calls the function, etc. It also explicitly lists information that is non-trivial to retrieve from the complex trace structure such as read/write access offset of a file.

PrismIO provides several analytical APIs that report aggregated information from different aspects, such as I/O bandwidth, I/O time, etc. It also provides feature extraction APIs that extract important I/O characteristics users may be interested in. These APIs will be part of our prediction workflow that we will discuss in later sections as they extract features needed for the model. In addition, PrismIO provides visualization of the trace. The following is a portion of frequently used APIs. Due to the space limit, we do not discuss all APIs in detail. Readers can check our GitHub repo for details if interested.

Feature extraction is a class of APIs that extract the I/O characteristics of an application. As introduced previously, one primary purpose of these APIs is to prepare inputs to the model. In the machine learning context, the inputs to a model are called features. As our model models whether to place files on BBs based on I/O characteristics of applications, I/O characteristics are the input features to our model. In the rest of the paper, when we discuss topics related to the model, we use I/O features to refer to I/O characteristics. Such features include I/O type (RAR, RAW, WAR, etc.), access pattern (random or sequential), file per process (FPP), I/O interface (POSIX, MPI-IO, etc.), etc. For an arbitrary application, users either need to read the source code or analyze the trace to get them. With the original trace, users may spend considerable time writing code to analyze it, while with PrismIO, users can get those features by function calls. We provide examples to demonstrate the simplicity of using them.

We use IOR runs that perform quite differently between PFS and BB from the experiment dataset introduced in Section IV as our use cases for analysis and call them extreme runs. Figure 9 plots the I/O bandwidth of IOR runs with 64 processes on 2 nodes on GPFS and BB on Lassen. Each point represents a run with a certain I/O feature combination. The x-axis is the I/O bandwidth on PFS and the y-axis is the I/O bandwidth on BB. The farther the point from the diagonal, the IOR run with that configuration performs more differently on PFS and BB.

We pick the highlighted runs in Figure 9 to conduct detailed analyses. The upper left one performs much better on BBs than on PFS. The lower middle one performs much better on PFS than on BB. We refer to them as “Good BB Bad PFS” and “Good PFS Bad BB” in later text. Since they perform very differently between systems, they may indicate bottlenecks of systems. We trace them with Recorder, and then do detailed analyses with PrismIO.

We run IOR with the same configuration but on more nodes. Figure 8 demonstrates the I/O timeline of IOR runs on 2, 4, 8, 16 nodes. We noticed there is one and only one node that performs much better than the other nodes, so we checked if it is the same one in these four runs. Unfortunately, we did not observe a common node. Therefore, we conclude that it’s a system problem of GPFS in balancing metadata operations like open. We checked with LC admins and confirmed the low-level reason is that GPFS does not have metadata or data servers. When writing a bunch of files to the same directory, one of the nodes becomes the metadata server for that directory and thus ranks on it are much faster than ranks on other nodes for metadata operations.

Figure 11 (c) demonstrates the I/O timeline of the same run on BB but with direct I/O enabled. All ranks now take the same long time on BBs. This confirms that the load imbalance over ranks on BBs is due to the SSD hardware cache. GPFS is a shared file system so its hardware cache is likely to be occupied by millions of other workloads, while the hardware cache of node-local BBs is not filled until the total amount of I/O exceeds the cache size. When the hardware cache is filled, ranks with I/O that cannot use the cache have to run much slower than ranks with I/O that can use the cache. Therefore, the empirical conclusion from some research that large I/O does not perform well on BB is not completely correct. A single large I/O can still perform better on BB than on GPFS if the BB hardware cache is not full. Users can still do large I/O on BB, as long as the total amount of I/O at a time does not overwhelm the hardware cache.

From the previous case studies, we identified some values of features that can be more preferable to BB. We saw MPI-IO, POSIX, transfer size, etc. can affect the choice of BB. Moreover, later in the model section, we get feature importance from the model. We will observe transfer size and read/write patterns are important features for the prediction. To give a summary of such trends, for a certain feature, we fix its value, and look at all runs with that feature equal to that value. We count how many runs are better on BB and PFS. Figure 12 demonstrates the percentage of runs better on Lassen BB as the values of features change. As transfer size grows, the percentage of runs better on BB decreases. Similarly, the percentage of runs better on BB is higher when reading than writing. We can conclude that although overall BB brings better performance than GPFS, it prefers smaller transfer sizes and read operations.

The key to answering whether to place files written by an application on burst buffers is to understand the relationship between the I/O characteristics of an application and it’s performance on different I/O subsystems. As defined previously, we use I/O features to refer to I/O characteristics that we use as inputs to our model. Since some parameters are continuous such as transfer size and number of read/write, we need a model to interpolate the relationship. And since the parameter space is large, heuristic-based methods may not work well and are difficult to be extended when adding new I/O features. Therefore, we decided to apply machine learning (ML) techniques to model such relationships. An application can have multiple files with very different I/O patterns. These files from the same application can usually be placed on different I/O subsystems. Therefore, our model focuses on individual files instead of the whole application. Ideally, it predicts the best I/O subsystem for each file and thus results in the best file placement plan globally for the application.

Since Lassen uses node-local BBs, the target is either using it or not. So it should be a binary classification problem. We evaluate several different classifiers and compare their test accuracies. We do 90-10 split on our dataset. We train the model with 90% of the data, and then use the rest 10%, which is unseen for the model, to get the test accuracy by comparing predictions with the true labels. To mitigate the variability from data split, we train and evaluate each classifier 10 times with random data split every time. We report the test accuracy of the classifiers we tried in the result section.

We make the selection process efficient by developing a workflow. This is because even though we obtained the approximation of the relationship with the ML model, applying it to large-scale HPC applications with a large number of different files is non-trivial. When users don’t know the I/O features of an application, they cannot apply the model. To solve this problem, we combine the feature extraction functions of PrismIO with the model and make them into a workflow. Figure 13 highlights the workflow design and demonstrates how parts fit together in the workflow and their inputs/outputs. Given an application, the user should only need to run it once on any machine and trace it with Recorder. Then the workflow would take the Recorder trace, extract I/O features for each file, and feed them into the pre-trained model to predict the best file placement, namely which file should be placed on which system for all files.

We have made the workflow a function called predict in PrismIO. Users simply need to call it with the trace path. It groups the data by file names and calls feature extraction APIs on each of them. It loads the model and predicts whether to use BBs for each file. The I/O features and predictions are organized as a dataframe indexed by file name. Figure 14 demonstrates a part of the workflow output of a sample trace.

We use the percentage of correct prediction to evaluate our classifiers. In other words, we compare the predictions with the ground truth and see what percentage it predicts the correct answer. We use accuracy to refer to this. For Lassen among the classifiers we tried, the Decision Tree classifier gives the highest test accuracy, which is 94.47% (Figure 15). Therefore, we decide to continue with the Decision Tree classifier.

The Decision Tree classifier has metrics for feature importance. Figure 16 demonstrates the importance of all features. Since our model predicts whether a combination of I/O features performs better on GPFS or BB, higher feature importance for model prediction implies the feature can be a key factor that distinguishes GPFS and BB. It may also imply underlying bottlenecks of a certain system. On the other hand, features with very low importance may be insignificant for prediction and accuracy. Therefore, features are selected after model selection by eliminating the least important features recursively until the accuracy significantly drops. After selecting the classifier and feature set, the chosen classifier is trained using the same data. This training is offline and the trained model can be loaded at runtime.

In section V, we presented our test of the model on unseen IOR runs and got a 94.47% accuracy for Lassen. However, IOR is not a real application and we directly know the I/O features from the configurations of IOR runs. To further evaluate how our prediction workflow works on real applications, we test it with four unseen applications: LAMMPS, CM1, Paradis, and HACC-IO. LAMMPS is a molecular dynamics simulator, CM1 is an extreme climate simulator, Paradis is a dislocation dynamics simulator, and HACC-IO is the I/O proxy application of HACC, which is a cosmology simulator.

First, we run these applications on both Lassen GPFS and BBs and trace them with Recorder. Although we only need the trace on any one of the systems to make the prediction, we need both to get the ground truth. We run them with configurations from example use cases provided in the documentation or repository of these applications. Second, for each run, we apply our prediction workflow on either the GPFS trace or BB trace. As discussed in section V, it predicts which I/O subsystem to use for each file. Third, we get the ground truth of whether to use BBs for each file by comparing its I/O bandwidth on GPFS and BBs. We apply the PrismIO io_bandwidth API discussed in section III to get the I/O bandwidths of each file. Finally, we compare the predicted system with the ground truth to get accuracy.

We get 121 samples in our test data set. Since our model predicts whether to use BBs for a file based on its I/O features, each individual file produced by runs should be a sample. However, although these applications write a large number of files, many of them have exactly the same I/O features. For example, one LAMMPS run in our experiment has 386 different files, but most of them are dump files. LAMMPS does multiple dumps and it writes files in the same way for each dump. Such files will have exactly the same I/O features and the prediction for them will also be the same, meaning they cannot be used as different samples. Therefore, we create more application runs by varying their input configurations such as problem size, the number of processes, interfaces, etc. They will have more files with different I/O features and thus produce more samples. We group files with the same I/O features and treat them as a single sample. We use average bandwidth to figure out the ground truth for the group.

Figure 17 demonstrates the test accuracy. We achieve 95.86% accuracy on the whole test data set. For individual applications, we experimented with seven different HACC-IO cases and achieved 100% accuracy because it has quite simple I/O patterns. We experimented with two Paradis cases, three CM1 cases, and three LAMMPS cases to cover different I/O features. We achieved 96.67%, 96.55%, and 92% accuracy for Paradis, CM1, and LAMMPS, respectively.