PROV-IO⁺: A Cross-Platform Provenance Framework for Scientific Data on HPC Systems

Data-driven scientific discovery has been well acknowledged as a new fourth paradigm of scientific innovation [1]. The shift toward the data-driven paradigm imposes new challenges in data findability, accessibility, interoperability, reusability (i.e., FAIR principles [2, 3]) and trustworthiness [4], all of which demand innovative solutions for modeling and capturing provenance, i.e., the lineage of data life cycle.

As an example, Figure 1 shows a simplified scientific workflow which analyzes geophysical sensing data on high performance computing (HPC) systems (i.e., DASSA [5]) . The workflow takes geophysical data as input, which are often stored in different file formats (e.g., “.tdms”, “.h5”). It then converts non-HDF5 files into a uniform HDF5 format (i.e., “.h5”). Depending on the analysis goals, the workflow further applies a set of analysis programs (e.g., “Decimate”, “X-Correlation-Stacking”) to process the files, the results of which are stored as data products in HDF5 format.

Based on our survey, the domain scientists using DASSA need the fine-grained origin of the data products (i.e., backward data lineage). For example, User A applies the “Decimate” program with a number of HDF5 files as input and generates a set of data products. Another User B may query the origin of the datasets in the final data products to understand which datasets in the input files contributed to which portions of the final data products, or who initiated the “Decimate” application to generate the data products and when. Such provenance information is important for ensuring the reproducibility, explainability, and security of the DASSA data.

Nevertheless, the DASSA workflow involves multiple programs accessing multiple files using different I/O interfaces and operations (e.g., HDF5 and POSIX), which makes tracking and deriving the data provenance non-trivial. Moreover, as we will elaborate in Section §3, there are other diverse needs of provenance for different scientific workflows and data (e.g., I/O statistics, configuration lineage). Such diversity, complexity, as well as the stringent performance requirement in HPC environments call for a practical solution beyond the state of the art.

Unfortunately, to the best of our knowledge, existing provenance tools cannot address the grand challenge above sufficiently due to a number of limitations:

First, while the importance of provenance has been well recognized across communities in general (e.g., databases [6, 7, 8, 9, 10], operating systems (OS)  [11, 12], eScience [13, 14, 15, 16, 17]), there is a lack of concrete understanding of the exact provenance needs of domain scientists, largely due to the variety of data and metadata that could be generated from HPC systems. As a result, existing solutions are often too coarse-grained (e.g., whole file tracking without understanding HPC data formats  [11]) to help domain scientists effectively, or too specific for one use case (e.g., Machine Learning [18]) to support general needs.

Second, in terms of provenance modeling, we find that existing standards (e.g., W3C PROV [19]) are too vague to describe the characteristics of scientific data provenance precisely. Scientists often seek a variety of information from scientific workflows on HPC systems, including the origins of data products, the configurations used for deriving results, the usage patterns of datasets, and so on, which cannot be described effectively using any existing provenance models. Such ambiguity limits the capability of existing provenance solutions for describing scientific data.

Third, in terms of usability, existing approaches often require the users to identify the critical code sites in the workflow software (e.g., loop structure [20]) and manually insert API calls to track the desired information accordingly. Moreover, they often rely on many external packages to work properly, which make it difficult to deploy and use them on different HPC platforms. The labor-intensive and error-prone approaches, together with the portability and compatibility issues, hinder the wide adoption of provenance products and diminishes the potential benefits.

Note that the limitations highlighted above are correlated. For example, the lack of understanding of provenance needs and the ambiguity of the provenance model are contributing to each other, which fundamentally limits the usability of existing solutions in terms of granularity, expressibility, etc., which in turn makes clarifying the ambiguity and real needs difficult.

We tackle the grand challenge of provenance support for scientific data on HPC systems in this paper.

First, we observe that for a provenance framework to be practical and useful, inputs from the end users (i.e., domain scientists) is essential. Therefore, we collaborate with domain scientists to analyze four representative scientific workflows in depth. In doing so, we identify the unique characteristics of the workflows studied (e.g., I/O interfaces, data formats, access patterns) as well as the specific needs for scientific data provenance (e.g., lineage at file, dataset, or attribute granularity).

Second, we observe that I/O operations are critically important in affecting the state of data that form the lineage needed by the domain scientists. Therefore, different from existing solutions [20, 21, 22], we introduce an I/O-centric provenance model dedicated for the HPC environments. The model enriches the W3C PROV standard [19] with a variety of concrete sub-classes, which can describe both the data and the associated I/O operations and execution environments precisely with extensibility. Moreover, it enables us to decouple the data provenance from specific executions of a workflow and support the integration of provenance from multiple runs naturally, which is important as workflows may evolve over time.

Third, based on the fine-grained provenance model, we find that the rich I/O middleware already used by the scientists provide an ideal vehicle for capturing the desired provenance transparently. Therefore, we create a configurable and extensible library and integrate it with existing I/O code paths (e.g., HDF5 I/O and POSIX syscalls) to capture necessary information without requiring the scientists to modify the source code of their workflows. Moreover, to further improve the usability, we persist the captured provenance as standard RDF triples [23] and enable provenance query and visualization.

Forth, through the communication with domain scientists in the industry, we notice the increasing importance of supporting containerization [24]. By wrapping the HPC workflows together with their dependencies in containers, the containerization techniques can effectively reduce the burden of software maintenance and thus enable more desired features including reproducibility, reusability, interoperability, etc. Therefore, a provenance framework should be generic enough to handle provenance in both containerized and non-containerized scenarios.

Based on the key ideas above, we build a framework called PROV-IO⁺, which can provide end-to-end provenance support for domain scientists with little manual effort across platforms. We deploy PROV-IO⁺ on representative supercomputers and evaluate it with realistic workflows. Our experiments show that PROV-IO⁺ incurs reasonable performance overhead and outperforms a state-of-the-art provenance product (i.e., IBM ProvLake [20]) for the use cases evaluated. More importantly, through the query and visualization support, PROV-IO⁺ can address the provenance needs of the scientists effectively.

In summary, we have made the following contributions:

• •

  Identifying concrete provenance needs of domain scientists based on four representative scientific workflows;

• •

  Designing a comprehensive PROV-IO⁺ model to describe the provenance of scientific data precisely and extensibly;

• •

  Building a practical prototype of PROV-IO⁺ which can support different HPC workflows with little human efforts in both containerized and non-containerized scenarios;

• •

  Measuring the PROV-IO⁺ prototype in HPC environments and demonstrating the efficiency and effectiveness;

• •

  Releasing PROV-IO⁺ as an open-source tool to facilitate follow-up research on provenance in general.

Experiments were performed on three state-of-the art platforms, including LBNL Cori Supercomputer [25], Samsung SAIT Supercomputer (SuperCom) [26], and Google Cloud Platform (GCP) [27]. First, in terms of non-containerized scenario, we applied PROV-IO⁺ to three scientific workflows (i.e., DASSA [5], Top Reco [28], and an I/O-intensive application based on H5bench [29]) on the Cori supercomputer. Second, in terms of containerized scenario, we applied PROV-IO⁺ to analyze one deep learning workflow (i.e., Megatron-LM [30]) on SuperCom. As will be discussed in §3, these use cases cover diverse characteristics (e.g., various languages, file formats, I/O interfaces, metadata) and provenance needs (e.g., file/dataset/attribute lineage, metadata versioning, I/O statistics). We varied the critical parameters of the workflows to measure the run-time performance and storage requirements under a wide range of scenarios. Third, we examined container’s impact on provenance tracking with Megatron-LM [30] on the Google Cloud Platform where we can schedule both containerized and non-containerized workflows and perform a fair comparision. In addition, we compared PROV-IO⁺ with ProvLake [20] using the Python-based Top Reco workflow as ProvLake only supports Python at the time of this writing. The PROV-IO⁺ prototype is open-source at https://github.com/hpc-io/prov-io.

The PROV family of specifications, published by the World Wide Web Consortium (W3C), is a set of provenance standard to promote provenance publication on the Web with interoperability across diverse provenance management systems [31]. One key specification is PROV-DM, an extensible relational model which describes provenance information with a graph representation. As shown in Figure 2, a W3C provenance graph abstracts information into classes of Entity, Activity, Agent, and Relation between the first three classes. Another critical specification is PROV-O which describes the mapping of PROV-DM classes to RDF triples. In PROV-O, Entity, Activity and Agent are mapped to subjects and objects, while Relation is mapped to predicates. We follow the W3C PROV standard in the PROV-IO⁺ design.

I/O libraries (e.g., ADIOS [32], HDF5 [33], and NetCDF [34]) play an essential role in scientific computations. Many workflows leverage the library I/O to manipulate data files. For example, HDF5 (i.e., Hierarchical Data Format version 5) is one of the the most wildly used I/O libraries for scientific data [35]. It is developed to be a parallel data management middleware to bridge the gap between HPC applications and the complicated, low-level details of underlying file systems, and has grown to a popular data format and management system.

In this work, we integrate our solution with the HDF5 library besides the classic POSIX I/O operations. This is based on the observation that HDF5 has evolved with a Virtual Object Layer (VOL) which can intercepts object-level API operations to functional plugins, called VOL Connectors [36]. VOL connectors allow third-party developers to add desired storage functionalities, which can be loaded dynamically at runtime. We leverage such extensibility for tracking the provenance of HDF5 I/O data.

In Figure 3, we show the key steps of the Top Reco workflow. First, the workflow takes two types of files as input, including the “.root” file for input event and the “.ini” file for configuration. Second, it generates “.tfrecord” files which stores the training dataset and test dataset based on the input events. Third, it trains a GNN model with the training dataset and tests the model with the test dataset by accessing the “.tfrecord” files. Fourth, a range of scores of edge and nodes are generated as the output of the model. Finally, a reconstructor component runs a simulation of reconstructing the top quarks based on highest scores. As summarized in Table I, the Top Reco workflow uses the POSIX I/O interface, and involves multiple programs accessing multiple files.

Moreover, with the growing popularity of container technologies [45, 46], HPC systems have started to be integrated with container-based job runtime tools [24]. In such HPC systems, large-scaled parallel pretraining models are executed in a containerized environment, which largely avoids tedious efforts in resolving installation or runtime dependencies (e.g, PyTorch [47] and nccl [48]). Besides the challenge of the Megatron-LM workflow itself, the domain scientists using the workflow would like to execute the workflow in a containerized HPC environment. Since a container is an isolated environment by design where application cannot directly interactive with the host system, how to containerized Megatron-LM and track provenance information of the containerized workflow at scale on HPC systems is another major challenge in this use case.

Note that both Megatron-LM and Top Reco ( §3.1) belong to deep learning provenance use cases. However, Megatron-LM has two major differences compared to Top Reco. First, in Megatron-LM, the provenance need is checkpoint-configuration consistency, while Top Reco’s provenance need is configuration version control. Second, Top Reco is a traditional single thread workflow, while Megatron-LM is a containerized parallel workflow, which introduces more challenges in terms of both provenance tracking and provenance storage.

By analyzing the four cases in depth and consulting with the domain scientists, we find that there is a big gap between the provenance needs and existing solutions. The variety of the workflow characteristics (e.g., different I/O interfaces and file formats) as well as the diversity of scientists’ needs motivates us to design a comprehensive provenance framework to address the challenge, which we elaborate in the following sections.

Figure 5(a) shows an overview of the PROV-IO⁺ model, which is derived based on the W3C standard (§2.1) as well as the characteristics of typical workflows and the provenance needs of domain scientists (§3).

Following the W3C specification, we classify information into five PROV-IO⁺ super-classes: Entity (yellow boxes in Figure 5(a)), Activity (purple boxes), Agent (orange boxes), Extensible Class (green boxes) and Relation (text on arrows). Moreover, we introduce a variety of concrete sub-classes to enrich the model, which can capture the data with different granularity as well as the associated I/O operations and execution environments for deriving the data. We summarize the definitions of the sub-classes in Table II and highlight the main concepts added to each super-class as follows:

Figure 6 shows the architecture of the PROV-IO⁺ framework, which supports two usage modes: Mode #1 (Figure 6a) provides provenance support for traditional non-containerized workflows on HPC systems; Mode #2 (Figure 6b) supports containerized workflows. There are five components in total, including: (1) the PROV-IO⁺ model (yellow) to specify the provenance information §4.1; (2) a provenance tracking engine (blue modules) which captures I/O operations from multiple I/O interfaces; (3) a provenance store (green) which persists captured provenance into RDF triples; (4) a user engine (red) for users to query and visualize provenance information; (5) a containerizer engine (purple) to support other components in containerized environments.

Among the five components, the PROV-IO⁺ model (yellow) has been discussed in details in §4.1. We introduce the other three common components used in both modes (i.e., provenance tracking, provenance store, and user engine) one by one in §4.2.1, and then discuss the containerizer engine for supporting containerized workflows in §4.2.2.

We apply PROV-IO⁺ to three representative non-containerized workflows including Top Reco [28], DASSA [5], and an H5bench-based workflow [29]. As mentioned in §3, the three use cases exhibit diverse characteristics (e.g., various file formats, I/O interfaces, metadata) and provenance needs (e.g., file/dataset/attribute lineage, I/O statistics, metadata versioning). We summarize the information tracked by PROV-IO⁺ in the experiments to meet the provenance needs in Table IV and elaborate them in detail in the following subsections.

We apply PROV-IO⁺ to the representative deep learning workflow Megatron-LM [30], which exhibits unique characteristics and provenance needs as discussed in §3.4 and summarized in Table IV. We containerize the workflow through the PROV-IO⁺ containerizer engine and leverage eight NVIDIA A100 GPUs on SAIT SuperCom to accelerate the training. For clarity, we present the evaluation results on SAIT SuperCom in §6.4.

More specifically, we use the GCP Deep Learning virtual machines (VMs) with 32 vCPUs, 120 GB DRAM, and 4 NVIDIA T4 GPUs for the comparison experiments. And we apply PROV-IO⁺ in two different modes for provenance tracking on two versions of Megatron-LM (i.e., non-containerized and containerized) respectively. We discuss the comparison results in §6.5.

In case of Top Reco, the scientists need the mapping between configurations and the training performance. Therefore, PROV-IO⁺ tracks three domain-specific items (e.g., model hyperparameters, dataset preselections, and training accuracy) based on the extensible class defined in the PROV-IO⁺ model. To track the mapping between workflow configuration and training accuracy, we instrument the workflow’s training loop with PROV-IO⁺ APIs and record the training accuracy at the end of each epoch, and add the training accuracy to the provenance graph as a property of configurations. In addition, we vary the number of training epochs to see how the performance scales. Note that Top Reco is a single process workflow.

Figure 7(a) shows the performance for Top Reco. The y-axis is the normalized completion time (starting with 0.998), while the x-axis is the number of training epoch (roughly equivalent to training time). The grey bars are the baseline without provenance, and the green bars show the performance with PROV-IO⁺ enabled. We can see that the tracking overhead is negligible overall with a maximum of 0.02%. The overhead with a shorter training time is relatively high, which is mostly caused by the latency of Redland. As the number of training epoch increases, the overhead of PROV-IO⁺ decreases almost linearly because PROV-IO⁺ tracks a constant amount of information.

In case of DASSA, the scientists need the backward lineage of data products in different granularity. As shown in the second column of Table IV, PROV-IO⁺ tracks the information of user, program, file, dataset, or attribute for different lineage needs based on the PROV-IO⁺ model (§4.1). We follow a similar configuration as the domain scientists’ by using 32 compute nodes and up to 2048 input files (1.35TB in total).

Figure 7(b) shows the tracking performance for DASSA. The x-axis means the number of input files; the y-axis on the left and right sides show the normalized completion time and the raw completion time (in second), respectively. The grey bars represent the normalized baseline without PROV-IO⁺, and the red, green and blue bars represent the normalized completion time under three usage scenarios (i.e., “File Lineage”, “Dataset Lineage” and “Attribute Lineage”) where different provenance granularity are enabled (e.g., for “File Lineage” we enable “program”, “I/O API” and “file” tracking). The solid grey line means the average baseline completion time (in second) without provenance tracking, while the dashed blue line represents the worst case raw completion time with PROV-IO⁺ under all scenarios.

We can see the max overhead occurred when tracking the attribute lineage of the entire 2048 files, which is about 11%. This is because DASSA heavily relies on HDF5 attributes. To access an attribute, the program first needs to open the file and the dataset containing it, which incurs more I/O operations to track. But overall, PROV-IO⁺ incurs reasonable overhead in DASSA (from 1.8% to 11%). This is expected because DASSA does not require heavy I/O API tracking. In other words, PROV-IO⁺ is efficient for tracking the backward lineage in file, dataset, and attribute granularity.

In the H5bench based workflow, the scientists need the data usage and I/O statistics in general. We consider three different usage scenarios with different needs. As summarized in Table IV, scenario-1 tracks the total number of I/O APIs; scenario-2 tracks both the I/O API count and their duration for bottleneck analysis; scenario-3 tracks the users and threads that modify the file. Moreover, for each scenario, we consider three different I/O patterns including: write-read, write-overwrite-read, and write-append-read. In (c) and (d), we run the workflow with 128 to 4096 MPI processes. In (e), since the append operations from a large amount of MPI processes can easily overwhelm the memory buffer for appending and lead to out-of-memory (OOM) errors, we reduce the number of MPI processes ( 2 to 64). Also, based on the observation that the computation time of many HPC applications may vary from dozens to thousands of seconds per I/O operation, we introduce a relatively modest computation time of 25 seconds per step in the experiments.

Figure 7 (c) (d) (e) show the tracking performance under three different I/O patterns (i.e., “write+read”, “write+overwrite+read”, “write+append+read”). The x-axis stands for the number of MPI ranks. The left y-axis is the normalized completion time and the right y-axis is the raw completion time in second. The grey bars represent the baseline while the three types of colored bars stand for the performance of different provenance usage scenarios mentioned in Table IV(red for “scenario 1”, green for “scenario 2”, blue for “scenario 3”). The grey solid line is the average baseline completion time, while the blue dash line is the worst-case raw completion time with PROV-IO⁺ enabled.

Overall, we find that PROV-IO⁺ incurs reasonable amount of overhead (i.e., ranging from 0.5% to 4%) even under heavy I/O operations (3.9TB data with 4096 MPI ranks). In particular, the PROV-IO⁺ overhead under the “write-append-read” I/O pattern (Figure 7 (c)) is minimal (around 0.5%). This is because the HDF5 I/O operation under this pattern takes more computation time than under the other two patterns to determine the append offset and memory range, which makes the PROV-IO⁺ overhead more negligible. Also, by comparing scenario-1 and scenario-2, we find that tracking the I/O API duration introduce little additional overhead. This is reasonable because the timing information can be piggybacked with the I/O API tracking which dominates the overall tracking time.

The storage requirement of PROV-IO⁺ is directly related to the amount and the class of information tracked. Specifically, the storage overhead may increase in two ways: (1) the size of a single provenance record may increase (e.g., adding timing information will increase size of an I/O API record); (2) the total number of records in a provenance file may increase (e.g., tracking thread information will create a number of thread records). We summarize the storage performance of PROV-IO⁺ for the three workflows in Figure 8.

Figure 8(a) shows the Top Reco case. The x-axis represents the number of epochs and the y-axis is the provenance size (KB). We can see that the provenance size is negligible. This is because PROV-IO⁺ allows users to specify the target provenance precisely without incurring unnecessary overhead. It also scales linearly since the number of new nodes added to provenance graph is the same as the increment in training epochs.

Figure 8(b) shows the DASSA case. The x-axis is the number of input files while the y-axis represents the provenance size (MB). Lines in three different colors represent File Lineage, Dataset Lineage and Attribute Lineage, respectively. We can see that the storage requirement varies from 40 MBs (with 128 input files) to about 800 MBs (with 2048 files) with linear scalability (note that the x-axis increases by a multiple of 2). Although DASSA heavily relies on attributes, the storage overhead in the three usage scenarios is similar. This is because I/O API is still the dominant part in all scenarios. Even though the number of file and dataset is far less than attribute in DASSA input data, when compared to number of APIs involved in the workflow, their contribution to storage overhead is insignificant.

Figure 8 (c)(d)(e) shows the H5bench-based workflow with three different I/O patterns. The x-axis represents number of MPI ranks and the y-axis stands for provenance size in MBs. Note that x-axis also increases by a multiple of 2. Lines in three different colors represents three different provenance usage scenarios (Table IV). We can see the provenance size varies from a few KBs to 168 MBs. Among the three I/O patterns, “write+overwrite+read” has the highest storage overhead under usage scenario 2. This is because the pattern includes one more I/O application (i.e., overwrite) than “write+read” and has much more MPI processes contributing to provenance graph than “write+append+read”. Moreover, scenario 2 also has the largest amount of tracked information (I/O API and their duration). Note that the storage overhead in this workflow also scales linearly.

In summary, because of the flexibility of the fine-grained PROV-IO⁺ model, PROV-IO⁺’s storage overhead is reasonable for all the use cases evaluated.

In this section, we introduce our experiments with a containerized workflow (i.e., Megatron-LM [30]) on the Samsung supercomputer (SuperCom).

Megatron-LM supports multiple pretraining models (§3.4), and we configure Megatron-LM to use GPT-2 as the pretraining model in this set of experiments based on the need of the domain scientists. The training dataset is WikiText103 [64] which is provided by the Megatron-LM authors [30]. We enable both model parallelism and data parallelism (§3.4) in the workflow for experiments.

As mentioned in §3.4, the major provenance need in the Megatron-LM use case is to ensure the consistency between the checkpoint and the workflow configurations. Therefore, we track detailed checkpoint information pertaining to a pretraining process (e.g., the path of the checkpoint file) as well as a variety of relevant configuration parameters (e.g., the number of GPUs, the batch size). The configuration information is tracked once at the beginning of workflow execution as the information remains invariant throughout the workflow execution, while the checkpoint information is tracked transparently by instrumenting PyTorch at the end of the workflow execution. In addition, we record the GPT-2 training loss at the end of each training iteration. We change the number of training iterations in the experiments to measure how the tracking performance and storage requirement scales. We report the measurement results as follows.

Figure 9a shows the provenance tracking performance. The y-axis is the normalized workflow completion time, and the x-axis is the number of training iteration. We use the grey bar to represent the baseline without provenance tracking, and the blue bar stands for workflow completion time with provenance enabled. For each experiment, we repeat it five times and calculate the average performance value. The result shows that the maximum tracking overhead is about 0.6% when the training iteration is set to 50. When the number of iteration increases, PROV-IO⁺’s tracking overhead tends to be negligible, which implies that PROV-IO⁺ is scalable in terms of tracking performance.

Figure 9b shows the storage requirement of tracking Megatron-LM with PROV-IO⁺. The y-axis is the provenance size (KB), and the x-axis is the number of training iteration. The result shows that, to track the checkpoint-configuration consistency information, the provenance size is negligible in general (e.g., less than 15 KB in all experiments). The size of the provenance information scales almost linearly as the number of training iterations increases, mainly because the training loss is recorded at the end of each training iteration.

In summary, to track the necessary provenance for maintaining the checkpoint-configuration consistency in containerized Megatron-LM, PROV-IO⁺ introduces small tracking and negligible storage consumption.

In this section, we analyze the impact of containerization on PROV-IO⁺’s provenance collection. As mentioned in §6.1, we leverage the GCP platform for the comparison experiments because we can setup different runtime environments for both containerized and non-containarized workflows on GCP. We apply PROV-IO⁺ in two different modes for tracking two versions of Megatron-LM (i.e., non-containerized and containerized) on GCP respectively.

To validate the impact of containerization, we execute the non-containerized version of Megatron-LM workflow and the containerized version separately on different GCP VMs. This is to ensure that there is no interference between the executions of the two versions. The provenance information tracked is the same as described in §6.4. We reduce the scale of the workflow to meet the VM’s resource constraints (e.g., vCPUs and memory).

The performance of PROV-IO⁺ on non-containerized Megatron-LM and containerized Megatron-LM are shown in Figure 10a and Figure 10b, respectively. In both cases, the y-axis is the normalized workflow completion time, and the x- axis is the number of training iteration. By comparing Figure 10a and Figure 10b, we can see that in both cases PROV-IO⁺ incurs little overhead, especially when the number of iterations is large. This suggests that containerization has little impact on PROV-IO⁺, and both modes of PROV-IO⁺ can support provenance tracking efficiently.

In conclusion, our experiments on three different platforms (i.e., Cori in §6.2, SuperCom in §6.4, and GCP §6.5) shows that PROV-IO⁺’s provenance tracking performance has little dependence on the platforms, and the overhead is consistently low across different execution platforms.

In this section, we compare PROV-IO⁺ to state-of-the-art provenance systems. Table V shows the basic characteristics of Komadu [65], ProvLake [20], and PROV-IO⁺. We can see that all three frameworks are derived from the base PROV-DM model, which makes the comparison fair. On the other hand, Komadu only supports Java programs and ProvLake only supports Python, which makes them incompatible with many C/C++ based scientific workflows (e.g., DASSA and H5bench). Note that PROV-IO⁺’s C/C++ interface is designed for integration with major HPC I/O libraries. Once the I/O library is integrated with PROV-IO⁺ (e.g., HDF5), the provenance support is mostly transparent to the workflow users, i.e., users can control the rich provenance features through a configuration file without manually modifying their source code with APIs. Neither Komadu nor ProvLake support such capability or transparency.

Since ProvLake has outperformed Komadu based on a previous study [17], we focus on the comparison with ProvLake. Because ProvLake does not support C/C++ workflows, we cannot apply it to DASSA and H5bench. Therefore, we compare the two provenance tools using Python-based Top Reco in the rest of this section.

Different from PROV-IO⁺ which is I/O-centric, ProvLake is ’process-oriented’. Specifically, ProvLake creates records based on the execution steps of a workflow, and the provenance data are maintained as attribute or property of individual steps. On the contrary, PROV-IO⁺ is not limited to the execution steps of the workflow. For example, it can track a task in the workflow, an I/O operation invoked by a task, a data object involved in the I/O operation, etc., all of which are further correlated via the relations defined by the PROV-IO⁺ model (§4.1). Such flexibility and richness is not available in ProvLake.

To make the comparison with ProvLake fair, we use the same instrument points in the Top Reco workflow for both tools. Specifically, we instrument Top Reco at its GNN training loop and track the training accuracy at the end of each epoch to corresponding provenance records. Since the workflow configuration is never changed during the entire workflow, we only add it to ProvLake’s record once at the beginning of the workflow. In addition, to be representative, we track three different numbers of configurations (i.e., 20, 40, and 80).

Figure 11(a),(b),(c) compares the provenance tracking performance of the two systems where y-axis is normalized completion time. Figure 11 (d),(e),(f) shows the storage overhead where y-axis is size in KB. In all figures x-axis is the number of configurations. In (a)(b)(c), grey bars stand for the baseline without provenance tracking, green bars show the normalized performance with PROV-IO⁺, and red bars show the performance with ProvLake. In (d)(e)(f), green lines stand for PROV-IO⁺ provenance file size and red lines stand for ProvLake provenance file size.

As shown in Figure11(a)(b)(c), both frameworks incur negligible tracking overhead (e.g., less than 0.025%) and the PROV-IO⁺ overhead is even lower than ProvLake for most cases. Similarly, as shown in Figure11(d)(e)(f), PROV-IO⁺ always incurs less storage overhead, regardless of the number of configuration fields tracked. This is mainly because ProvLake has to track more irrelevant workflow information not needed in the use case.

As mentioned in §5, PROV-IO⁺ supports provenance query with visualization. Table VI summarizes the queries used to answer the diverse provenance needs of the three workflow cases. We can see that the provenance can be queried effectively and efficiently using a few simple SPARQL statements in general. Since the number of queries involved is small, the query time overhead is negligible in our experiments. We discuss each case in more details below.

In DASSA, to get the backward lineage of a data product, we can start with the program which generated the data product and look for its input data. The same procedure can be repeated as needed. For example, DASSA may convert “WestSac.tdms” into “WestSac.h5” with program “tdms2h5”, and then use “decimate” to process “WestSac.h5” into data product “decimate.h5”. To get the backward lineage of “decimate.h5”, in the query, we first retrieve with keywords “decimate.h5 prov:wasAttributedTo ?program” to locate program “decimate”. Next, in the same query, we add statement “?file wasAttributedTo ?program” to retrieve that program’s input file “WestSac.h5” which is the first level predecessor of “decimate.h5”. We can further expand the query by adding similar statements to locate “decimate.h5”’s earlier predecessors (e.g., “WestSac.tdms”).

As summarized in Table VI, for each backward step, we only need three query statements. Figure 12 shows the visualization of this example, which follows the PROV-IO⁺ provenance model (§4.1) and highlights the queried data lineage in blue. Other types of lineages (e.g., dataset and attribute) can be queried and visualized in the same way.

Similarly, in H5bench, we have three types of provenance needs (i.e., the scenarios described in §6.2) which can be answered using 1, 2, 3 SPARQL statements respectively. In Top Reco, the metadata versioning and mapping information can be queried in 2 statements. Note that the provenance needs are diverse across the real use cases, but the number of queries needed is consistently small. This elegant result suggests that PROVI-IO⁺ is effective for scientific data on HPC systems.

In case of Megatron-LM, users want to identify the checkpoint which is consistent with the configuration for the follow-up training process. Figure 13 shows a scenario where there are two different batch sizes used during the previous pretraining (i.e., “Batch_Size_A” is 128, and “Batch_Size_B” is 256), and there are three checkpoints generated based on the two batch sizes (i.e., “Checkpoint_1”, “Checkpoint_2”, “Checkpoint_3”). Assume the user wants to continue a GPT pretraining process which has a batch size of 256 with one of the existing checkpoints (i.e., “Checkpoint_3”), s/he can query the provenance with as few as 2 lines of SPARQL statements, as shown in the last row of Table VI. Moreover, the user can also add advanced conditions to the query to filter out the feasible checkpoint with the best quality (e.g., a checkpoint with certain training loss).