BAFLineDP: Code Bilinear Attention Fusion Framework for Line-Level Defect Prediction

Code preprocessing [27] plays a pivotal role within the realm of deep learning. It serves as a critical step in enhancing the robustness and stability of models by standardizing and normalizing the source code. By eliminating redundant information that holds no relevance to code logic, the impact of noise on the model is reduced, allowing the model to extract code semantics and syntactic features more effectively. Therefore, we performed code preprocessing for each code file. Based on the investigation by Rahman et al. [28], we systematically removed all special characters (e.g., (),.:;’!”(space)) since these special characters introduce undesirable noise to the prediction model. Simultaneously, blank lines were also removed because of no substantive contribution to the code’s behavior [29]. Eliminating such extraneous information allows the model to maintain its focus on learning the actual content of the code, free from interference by irrelevant factors. In addition, to enhance the model’s generalization capability, we introduce the generic tokens $\langle$str$\rangle$ and $\langle$num$\rangle$ to replace constant strings and numeric representations within the source code. This strategic substitution ensures the model does not generate independent representations for these elements, rendering it more adaptable to diverse scenarios and usage cases.

In our pursuit of capturing precise semantics related to defective code, we utilize the pre-trained CodeBERT model to encode code lines. CodeBERT [30] is a bimodal pre-trained language model that leverages a multi-layer bidirectional Transformer as the architecture to capture the meaningful semantic relationships among diverse tokens, contributing to construct an adequate representation of the source code line. The incorporation of the pre-trained CodeBERT model holds the promise of a deeper understanding of code line semantics and the accurate capture of features associated with defects. Furthermore, to maintain the structural information of the source code, we adopt a sequential structure to depict the organization of code files. In this configuration, each code file is expressed as a sequence of code lines, denoted as $\langle l_{1},l_{2},\cdots,l_{n}\rangle$. For each code line, the pre-trained CodeBERT model will generate the corresponding embedded representation.

In this step, we employ the Bi-GRU network to extract line-level contextual features, thereby capturing the global information pertaining to code lines. The Bi-GRU [12] is founded upon a bidirectional recurrent neural network architecture, which affords it the capability to concurrently consider both the preceding and succeeding information within sequences of source code lines. The bidirectional mode ensures the effective modeling of the contextual associations among code lines, culminating in a richer representation of their interdependencies.

Given a sequence of code lines, denoted as $V_{l}=[v_{l1},v_{l2},\cdots,v_{ln}]$, where $v_{li}\in R^{1\times d}$ represents the vector representation of code line $l_{i}$, $i\in[1,n]$. These representations are encoded by the pre-trained CodeBERT model, where $d$ signifies the output dimension of CodeBERT. Bi-GRU comprises both a forward GRU denoted as $\overrightarrow{h}_{i}=\overrightarrow{GRU}(v_{li})$, $i\in[1,|n|]$ and a backward GRU denoted as $\overleftarrow{h}_{i}=\overleftarrow{GRU}(v_{li})$, $i\in[|n|,1]$. Through the concatenation of the two hidden states, $\overrightarrow{h}_{i}$ and $\overleftarrow{h}_{i}$, produced by the forward and backward GRU components, we arrive at the final hidden representation of the given $v_{li}$. In other words, the line-level context vector representation can be expressed as $v_{ci}=[\overrightarrow{h}_{i}\oplus\overleftarrow{h}_{i}]$. Consequently, the line-level context sequence is represented as $V_{c}=[v_{c1},v_{c2},\cdots,v_{cn}]$, where $v_{ci}\in R^{1\times d^{{}^{\prime}}}$ characterizes the line-level context vector representation of code line $l_{i}$, and $d^{{}^{\prime}}$ signifies the output dimension of Bi-GRU.

Source code is inherently complex and structured [31], encompassing not only overall structure and semantic features but also local intricacies and contextual dependencies. Merely considering line-level context information is insufficient for a comprehensive understanding of defective code lines. Hence, to address this limitation, we introduce a bilinear attention mechanism [32] to build a code fusion network, called BAFN. BAFN is designed to amalgamate global and local information by capturing the bilinear interaction attention weights between code lines and their respective line-level contextual information to construct defect code features. The structure of BAFN, as depicted in Figure 3, primarily includes two essential modules: the bilinear interaction module and the bilinear pooling module.

Bilinear Interaction Module. Assume that the code line matrix denoted as $H_{l}\in R^{\theta_{l}\times d}$ and the line-level context matrix denoted as $H_{c}\in R^{\theta_{l}\times d^{{}^{\prime}}}$ are constructed by splicing the vectors from the code line sequence $V_{l}$ and the line-level context sequence $V_{c}$, respectively. $\theta_{l}$ represents the total number of code lines. We construct the bilinear interaction matrix $A\in R^{\theta_{l}\times\theta_{l}}$ through the source code line matrix $H_{l}$ and the line-level context matrix $H_{c}$. The calculation formula is as follows:

where $U\in R^{d\times k}$ and $M\in R^{d^{{}^{\prime}}\times k}$ are two learnable linear transformation matrices, which are also weight matrices. $k$ represents the dimension of the linear transformation, $q\in R^{1\times k}$ is a learnable weight vector. The symbol $\circ$ denotes the Hadamard product operation [33], and $\sigma(\cdot)$ signifies the ReLU activation function [34]. Each element within the bilinear interaction attention matrix $A$ encapsulates the extent of interaction between a source code line and its corresponding line-level context, further presenting the local associations between potential defects and code lines. In addition, to prevent the model from excessively focusing on extraneous noise information, we incorporate the mask into the interaction attention map $A$ to enhance the overall model performance.

Bilinear Pooling Module. To efficiently amalgamate global and local information from the code lines and create a high-level representation of defect code, denoted as $f^{{}^{\prime\prime}}\in R^{1\times k}$, we employ a bilinear pooling module. This module is instrumental in extracting important features from the code. The calculation formula for $f^{{}^{\prime\prime}}$ is as follows:

Notably, there are no new learnable parameters introduced in this layer. Instead, the bilinear pooling module and the bilinear interaction module share the weight matrices $U$ and $M$, thereby diminishing the number of model parameters. Furthermore, we add a sum pooling operation, grounded in the fused feature representation, to obtain a more compact defect code feature representation, denoted as $f^{{}^{\prime}}\in R^{1\times k/s}$. The calculation formula for $f^{{}^{\prime}}$ is as follows:

where the $SumPool(\cdot)$ function [35] is a one-dimensional non-overlapping sum pooling operation with stride $s$, which reduces the dimensionality of $f^{{}^{\prime\prime}}$ from $k$ to $k/s$, allowing for the extraction of crucial features. For the pooling operation, we set the stride $s=3$.

Further, we extend the single-head bilinear interaction attention matrix into a double-head form. The final defect code fusion feature vector $f\in R^{1\times k/s}$ is a combination of two heads: $f_{1}^{{}^{\prime}}\in R^{1\times k/s}$ and $f_{2}^{{}^{\prime}}\in R^{1\times k/s}$. The calculation formula for $f$ is as follows:

Since the shared utilization of weight matrices $U$ and $M$ across the two modules, introducing an additional header only requires adding a new weight vector $q$, which is parameter-efficient. Through our experiments, we find that incorporating a double-head interaction mechanism yields better performance when compared to the single-head interaction approach.

Via the BAFN, we derive a high-level vector representation $f$ of the source code file, which serves as a feature for performing file-level defect prediction. We employ a single-layer fully connected network as the prediction layer and utilize the source code file vector as input, leading to the generation of the defect prediction probability, denoted as $p$. The calculation formula is provided below:

where $W_{0}$ and $b_{0}$ are learnable weight matrices and bias values, and the $Sigmoid(\cdot)$ function [36] serves to map the prediction score into the $(0,1)$ range.

To identify defect-prone lines of code, we leverage the diagonal elements within the bilinear interaction attention maps to rank the risk associated with each source code line. We first consolidate the bilinear interaction attention maps from a double-head form into a single-head form. Then, we extract the diagonal elements within the consolidated map to assemble the line-level attention map. The attention score on the diagonal of this map is utilized as the final risk coefficient for each code line, which enables the sorting of all the code lines within the source code file. This sorting process allows us to pinpoint the most defect-prone code lines, ultimately achieving line-level defect prediction.

In our study, we utilize the line-level defect prediction datasets collected by Wattanakriengkrai et al. [9], encompassing 9 open-source projects and 32 software releases. We downloaded each project version from official websites for analysis, as detailed in Table I. This table includes project names, descriptions, release versions, file counts, line counts, file defect rates, and line defect rates. The datasets feature projects with a wide range of sizes and defect rates, with file counts varying from 731 to 8846, and line counts from 74k to 567k. File defect rates span 2% to 28%, and line defect rates range from 0.03% to 2.90%. For detailed steps on collecting file-level and line-level ground truth, please refer to [11]. Our experimental approach involves training on the first release of each project, validation on the second, and testing on subsequent versions. This results in 14 distinct tasks for within-project defect prediction (WPDP) and 112 for cross-project defect prediction (CPDP).

We adopt the classification metrics Area Under Curve (AUC) [37], Balanced Accuracy (BA) [38], and Matthews Correlation Coefficient (MCC) [39] to evaluate the file-level defect prediction task, and the workload-aware metrics Recall@Top20%LOC [11] and Effort@Top20%Recall [11] to evaluate the line-level defect prediction task.

AUC measures the performance of binary classification models. The AUC metric is unaffected by class distribution and, therefore, has better evaluation performance in the case of class imbalance.

BA measures the average ratio of true positive and true negative. The highly balanced accuracy shows that the method can accurately predict defective and clean instances.

MCC is used to evaluate the accuracy and stability of model predictions. MCC ranges from -1 to 1, where 1 means the predictions are perfect, 0 means the predictions are irrelevant to the actual outcome, and -1 means the predictions are completely opposite.

Recall@Top20%LOC is a metric utilized to gauge the efficacy of identifying defective lines within the top 20% LOC in a software release. A high value of Recall@Top20%LOC signifies the method’s proficiency in uncovering numerous actual defective lines with limited effort, whereas a low value of Recall@Top20%LOC indicates that developers need to spend more effort to detect defective lines.

Effort@Top20%Recall is a metric assessing the amount of effort (i.e., LOC) required to identify the top 20% of actual defective lines in a software release. A low Effort@Top20%Recall value suggests minimal effort is required by developers to pinpoint the top 20% of actual defective lines, while a high Effort@Top20%Recall value implies a greater effort is needed for the same task.

To evaluate the performance of our BAFLineDP approach, we compared the current advanced defect prediction methods. Below, we present four file-level defect prediction methods [40, 41, 42, 43]:

• •

  DBN, a deep belief network, automatically extracts semantic and structural code features to predict defective code files. Adhering to the experimental setup outlined by Wang et al. [40], our configuration comprises a batch size of 32, 10 hidden layers with 100 nodes each, an embedding dimension of 50, a learning rate of 0.01, and 200 training epochs for each hidden layer.

• •

  CNN is a convolutional neural network that automatically learns code semantics and context to predict defective code files. Aligning with the experimental setup of Li et al. [41], our configuration includes a batch size of 32, an embedding dimension of 50, a filter length of 5, 100 filters, a learning rate of 0.001, and 10 training epochs.

• •

  BoW (also known as Bag-of-Words), which leverages code token frequencies as features for source code files, is employed to predict defective code files. We align with the experimental deployment of Hata et al. [42], incorporating the SMOTE technique to address the class imbalance within training data. A logistic regression classifier is trained on code token frequency to construct the BoW model.

• •

  Bi-LSTM, a bidirectional long short-term memory network, is utilized to incorporate both past and future code information, enabling the automatic learning of semantic and syntactic features for predicting defective code files. We follow the experimental configuration introduced by Dam et al. [43], featuring a batch size of 16, an embedding dimension of 50, 64 nodes in the hidden layer, a learning rate of 0.001, and a training epoch of 50.

Subsequently, we introduce three advanced line-level defect prediction approaches [11, 44, 45]:

• •

  DeepLineDP, the state-of-the-art line-level defect prediction method, leverages a hierarchical Bi-GRU network to automatically capture context from surrounding tokens and code lines, enabling the extraction of code semantic features for file-level defect prediction. In addition, it incorporates a token-level attention layer to identify defective lines based on key tokens instrumental in file-level defect prediction. Our configuration, in line with the experimental setup of Pornprasit et al. [11], involves a batch size of 32, a learning rate of 0.001, an embedding dimension of 50, a single hidden layer with 64 nodes, a dropout ratio of 0.2, and 10 training epochs.

• •

  ErrorProne, an open-source tool by Google, leverages the Java compiler for static analysis and error detection in Java code, utilizing a set of error-prone rules to identify and rectify errors. By emulating developers’ sequential reading, ErrorProne analyzes code files top-down to pinpoint potentially defective lines. Adhering to the usage guide by Aftandilian et al. [44], we invoke the ErrorProne plug-in within a Java8 compilation environment for line-level defect prediction on the test code files.

• •

  N-gram is employed to assess the unnaturalness of code tokens by computing the entropy score of each token. Research conducted by Ray et al. [26] reveals that defect-prone codes tend to exhibit higher entropy values, indicating increased unnaturalness. Therefore, N-gram has predictive utility in identifying defective code lines. Following the experimental setup outlined by Hellendoorn et al. [45], we train a cache-based N-gram model using clean code files and then perform line-level defect prediction on defective code files. By calculating and ranking the average entropy score of tokens within each line, the most defect-prone code lines could be found.

It should be noted that DeepLineDP is categorized as a line-level defect prediction method due to its hierarchical approach to constructing code features, which incorporates information from surrounding code lines. This design enhances the method’s accuracy in identifying line-level defects, exhibiting state-of-the-art (SOTA) performance. However, we also consider involving DeepLineDP in file-level defect prediction for a comprehensive comparison with our BAFLineDP.

Our BAFLineDP model is implemented using PyTorch, a deep learning framework, and executed on a server equipped with an NVIDIA RTX 3090 GPU with 24 GB memory.

Regarding the experimental strategy, we employ binary cross-entropy as the loss function, utilize the Adam optimizer [46], and incorporate random dropout (rate of 0.2) and normalization to prevent model overfitting. Additionally, we address dataset class imbalance through weighted loss.

In terms of experimental parameters, we set the batch size to 16 and use the learning rate of 0.001. The model is trained at 10 epochs. The maximum input length of the pre-trained CodeBERT model is 75. The Bi-GRU network is configured with one hidden layer, while the number of nodes in the hidden layer is set to 64. The output dimension for the BAFN is designated as 256. Furthermore, the convolution kernel size of the bilinear pooling layer is 3.

About evaluation models, we consider the training model that attains the highest AUC value on the validation set as the ultimate evaluation model to perform both file-level and line-level defect prediction tasks.

Given the potential for certain datasets to yield models that exhibit significant performance discrepancies, we employ the Scott-Knott Effect Size Difference (Scott-Knott ESD) test [47] to compare the efficiency of different methods in our experiments. The Scott-Knott ESD test, a mean comparison technique, uses hierarchical clustering to categorize measurements (e.g., AUC) into statistically distinct groups with non-negligible effect size differences. It comprises two steps: (1) correcting the non-normal distribution of the input dataset and (2) merging any two statistically distinct combinations with negligible effect sizes into a single group. The rankings generated by the Scott-Knott ESD test ensure that (1) the differences in distribution magnitudes within each ranking category are negligible and (2) the differences in distribution magnitudes between ranks are not negligible. For a detailed description of the Scott-Knott ESD test, please refer to [47].

To assess the efficacy of our BAFLineDP approach, we conduct both file-level and line-level defect prediction in WPDP and CPDP scenarios, respectively, and compare with current advanced defect prediction techniques. Below, we present the approaches and results of two research questions (RQs).

(RQ1) What is the performance and cost-effectiveness of BAFLineDP in WPDP scenario?

Motivation. WPDP aims to identify and rectify software defects in their early stages, optimizing Software Quality Assurance (SQA) resource allocation and enhancing software quality. File-level and line-level defect prediction are techniques employed to realize these overarching objectives. Existing advanced defect prediction methodologies predominantly center on the file level, with limited attention to line-level approaches. Recently, DeepLineDP, which automatically learns the code hierarchical structure and considers code context, has been proposed to achieve SOTA performance in both file-level and line-level defect prediction within the WPDP scenario. However, this method disregards the importance of the context and local interaction information for code lines within line-level defect prediction. Therefore, we investigate whether BAFLineDP outperforms advanced file-level and line-level defect prediction methods within the WPDP scenario while offering superior cost-effectiveness.

Approach. To answer this question, we selected 14 training-validation-testing task combinations within the WPDP scenario. For instance, training and validation were conducted on datasets such as Hive-0.9.0 and Hive-0.10.0, followed by testing on Hive-0.12.0. Subsequently, we assess the performance of the BAFLineDP method by conducting evaluations in the context of both file-level and line-level defect prediction. In order to provide a comprehensive analysis, we compare BAFLineDP with four advanced file-level defect prediction approaches [40, 41, 42, 43], namely DBN, CNN, BoW, and Bi-LSTM. Additionally, we contrast BAFLineDP with three advanced line-level defect prediction methods [11, 44, 45], which include DeepLineDP, ErrorProne, and N-gram.

To evaluate the performance of these methods, we employ three traditional evaluation metrics (i.e., AUC, MCC, and BA) for file-level defect prediction and two effort-aware evaluation metrics (i.e., Recall@Top20%LOC and Effort@Top20%Recall) for line-level defect prediction. To effectively demonstrate the statistical performance disparities between different approaches, we apply the Scott-Knott ESD test. Figures 4 and 5 present the Scott-Knott ESD rankings and the distributions of corresponding metrics for BAFLineDP and other advanced file-level and line-level defect prediction approaches within the WPDP scenario.

Results. In light of the findings presented in Figure 4, our BAFLineDP method demonstrates notably superior performance, as evidenced by its average AUC, BA, and MCC values, which respectively stand at 0.793, 0.655, and 0.212. These values exhibit a significant advantage, ranging from 2% to 34%, 1% to 25%, and 23% to 1016% when compared to those of other file-level defect prediction approaches. The outcome indicates the superiority of our BAFLineDP over existing file-level defect prediction methods. In addition, the Scott-Knott ESD test also confirmed that BAFLineDP consistently ranks among the best in terms of AUC, BA, and MCC, which shows that the performance difference is statistically significant with a non-negligible effect size.

Notably, while BAFLineDP’s median BA is slightly lower than DeepLineDP’s, BA mainly assesses the precise differentiation between correct and defective files. However, in real-world software development, developers often favor a probabilistic ranking of defective files over a binary classification at a fixed threshold (e.g., 0.5). Therefore, when defective files are ranked based on probabilities, the superior performance of BAFLineDP is further confirmed by AUC, which assesses the ability to discriminate between defective and clean files.

Based on the outcomes illustrated in Figure 5, our BAFLineDP method exhibits an average Recall@Top20%LOC value of 0.395 and an average Effort@Top20%Recall value of 0.2. Compared to other line-level defect prediction approaches, BAFLineDP demonstrates a notable enhancement in cost-effectiveness, boasting an improvement of 19%-42% in accurately identifying defective code lines with a fixed 20% of the overall effort. Moreover, from a cost perspective, there is a reduction ranging from 10% to 51%. These findings firmly establish the superiority of BAFLineDP over existing line-level defect prediction approaches. Furthermore, the Scott-Knott ESD test also affirms that BAFLineDP consistently ranks among the top performers in terms of Recall@Top20%LOC and Effort@Top20%Recall, which shows that the performance difference is statistically significant with non-negligible effect size.

Answer to RQ1: The experimental findings within the WPDP scenario illustrate the superior performance of BAFLineDP in both file-level and line-level defect prediction compared to existing defect prediction approaches, achieving better cost-effectiveness and lower cost overhead.

(RQ2) How does BAFLineDP perform in both performance and cost-effectiveness within the CPDP scenario?

Motivation. The majority of existing defect prediction methods are primarily assessed within the WPDP scenario, yielding favorable performance. However, practical applications often encounter challenges such as limited availability of training samples for new projects and substantial variations in project structures and complexities [48]. Thus, although certain approaches excel in the WPDP scenario, their performance may not translate equally well to the CPDP scenario. Consequently, we focus on evaluating the performance and cost-effectiveness of BAFLineDP within the CPDP scenario while conducting a comparative analysis against other advanced file-level and line-level defect prediction methods.

Approach. To address this inquiry, we selected 112 training-validation-testing within the CPDP scenario. For instance, we utilize Hive-0.9.0 and Hive-0.10.0 for training and verification, respectively, while testing is conducted on JRuby-1.5.0. Similar to WPDP, we evaluate the performance of BAFLineDP in file-level and line-level defect prediction using three traditional metrics (i.e., AUC, BA, and MCC) and two effort-aware metrics (Recall@Top20%LOC and Effort@Top20%Recall). Furthermore, we compared BAFLineDP against four file-level defect prediction methods (i.e., DBN, CNN, BoW, and Bi-LSTM) and three line-level defect prediction approaches (i.e., DeepLineDP, ErrorProne, and N-gram).

It is important to note that ErrorProne, an open-source static code analysis plug-in, is unaffected by training data in line-level defect prediction. Its performance solely relies on customized code rules. Consequently, the performance results of ErrorProne in CPDP line-level defect prediction align with those observed in WPDP line-level defect prediction.

To provide a comprehensive and statistically significant analysis of performance differences among the approaches, we again employ the Scott-Knott ESD test. Figures 6 and 7 illustrate the Scott-Knott ESD rankings and the distributions of corresponding indicators for BAFLineDP and other file-level and line-level defect prediction methods within the CPDP scenario.

Results. According to the results in Figure 6, the average AUC, BA, and MCC values of BAFLineDP are 0.789, 0.647, and 0.212, respectively, outperforming other file-level defect prediction methods by 7%-46%, 5%-32%, and 91%-2027%. These results establish BAFLineDP’s superiority in the CPDP scenario. Additionally, the Scott-Knott ESD test confirmed BAFLineDP’s consistent top performers in AUC, BA, and MCC, which indicates that the performance difference is statistically significant with a non-negligible effect size.

As shown in Figure 7, the average Recall@Top20%LOC and Effort@Top20%Recall of BAFLineDP are 0.397 and 0.201, respectively. Compared to other line-level defect prediction methods, BAFLineDP exhibits a 12% to 42% improvement in identifying defective code lines using only 20% of the overall effort and a 14% to 51% reduction in costs. These findings position BAFLineDP as a superior choice over existing line-level defect prediction methods within the CPDP scenario. Furthermore, the Scott-Knott ESD test further confirms BAFLineDP’s top performance in Recall@Top20%LOC and Effort@Top20%Recall, indicating a statistically significant difference with non-negligible effect sizes.

Answer to RQ2: The experimental results illustrate that within the CPDP scenario, BAFLineDP outperforms existing defect prediction methods in both file-level and line-level defect prediction, while exhibiting higher cost-effectiveness and incurring lower cost overhead. These outcomes further reaffirm the effectiveness of BAFLineDP.

We conducted an ablation study with the principal objective of dissecting the individual contributions of pivotal modules within the BAFLineDP framework, discerning their respective impacts on the overall efficacy. Our scrutiny is particularly centered on the examination of three pivotal components, namely CodeBERT, Bi-GRU, and BAFN, given our conviction that they assume a pivotal role in the capture of essential facets pertaining to source code line defect semantics, global line-level context, and local interaction information of code lines. Furthermore, we chose to focus our ablation study on line-level defect prediction, as it holds greater practical applicability compared to file-level defect prediction.

Within the context of our ablation study, we observe the changes in the average Recall@Top20%LOC and Effort@Top20%Recall metrics resulting from the elimination or replacement of specific components, both in the WPDP and CPDP scenarios. The results of this ablation study are tabulated in Table II, wherein Abl. represents the performance of the simplified iteration of BAFLineDP, while Diff. signifies the performance version in comparison to BAFLineDP.

Replacing CodeBERT with Doc2Vec: In our investigation of CodeBERT’s impact, we substitute it with Doc2Vec [49], an unsupervised algorithm adept at acquiring fixed-length feature representations from variable-length textual data, for encoding lines of code. As demonstrated in Table II, after replacing CodeBERT, the cost-effectiveness of line-level defect prediction in WPDP and CPDP dropped by 12.4% and 15.4%, respectively, while the cost overhead increased by 11.0% and 13.9%, respectively.

Removing Bi-GRU: In exploring the role of Bi-GRU, we eliminate the utilization of Bi-GRU, thereby discontinuing the extraction of contextual information from code lines. As presented in Table II, the removal of Bi-GRU led to a decline in cost-effectiveness for line-level defect prediction in both WPDP and CPDP scenarios, registering reductions of 7.6% and 3.8%, respectively, while the cost overhead experienced a rise of 5.5% in WPDP and 1% in CPDP.

Removing BAFN: To investigate the significance of BAFN, the component responsible for local interaction information between source code lines and their contextual counterparts, we remove BAFN. As depicted in Table II, the removal of BAFN resulted in a marked reduction in cost-effectiveness for line-level defect prediction in WPDP and CPDP scenarios, with drops of 8.1% and 13.6%, respectively. In parallel, the cost overhead exhibited a substantial escalation of 27.5% in WPDP and 29.4% in CPDP.

In summary, CodeBERT, Bi-GRU, and BAFN each play pivotal roles in capturing essential elements of source code line defect semantics, global line-level context, and local interaction information of code lines. This contribution significantly enhances the cost-effectiveness of SQA resource management, concurrently alleviating the burden on developers by reducing the need for manual inspection of defective code lines and the associated workload.

In this section, we describe several threats that may have an impact on the effectiveness of our approach.