MVD: A Multi-Lingual Software Vulnerability Detection Framework

In recent years, learning-based approaches have been widely used to automate the identification/prediction of SVs in source code [18, 19]. The predictions have been performed on various levels of granularity, ranging from package/file to function and line. Among these levels of granularity, the function level is the most investigated as it reduces inspection effort for developers while still providing sufficient context of code for prediction [20, 11]. Thus, the function-level prediction is also adopted for our multi-lingual investigations.

Deep Learning has been increasingly investigated for function-level vulnerability prediction [21]. Recurrent Neural Networks like Long-Short Term Memory have been initially used for the task because of their ability to capture long-term dependencies in code (e.g., [9, 10, 8]). Later, to more precisely capture the structure and semantic meaning of code, graph-based models, including Gated Graph Neural Networks employed in Devign [22], ReVeal [7] or Graph Convolutional Networks used in IVDetect [23], Graph Attention Network in LineVD [20], have been explored for function-level vulnerability prediction. These graph-based models have shown superior performance than LSTM. Recently, LineVul [11] relying on CodeBERT [17], a pre-trained large language model, has demonstrated the state-of-the-art performance for function-level vulnerability prediction [24], outperforming various recurrent and graph-based neural networks. It is important to note that LineVul has only been evaluated on C/C++ vulnerabilities, and thus, its ability to perform multi-lingual vulnerability prediction is still largely unknown.

As mentioned in Section II-A, function-level vulnerability prediction has gained significant traction in the recent literature, but the latest advances, particularly using Deep Learning, for this task have mostly focused on detecting vulnerabilities in C/C++. However, we argue that multi-lingual vulnerability prediction is crucial in modern software development because of the following three reasons. Firstly, many large and widely used software systems are not written only in C/C++. For example, many mobile apps are written in Java; modern web development mostly requires JavaScript and TypeScript; Artificial Intelligence-based systems are frequently written in Python; the development of video games heavily relies on C#. Secondly, contemporary software projects are increasingly complex, often incorporating multiple programming languages to leverage the strengths of each, a.k.a. polyglot projects [13]. Specifically, Mayer et al. [25] found that polyglot projects are prevalent in practice, averaging five languages per project. This finding was later confirmed in a follow-up study through a survey with 139 software professionals [26]. Thirdly, the most dangerous vulnerabilities according to the top-25 CWE-IDs list in 2023,²²2https://cwe.mitre.org/top25/archive/2023/2023_top25_list.html are mostly language agnostic, except for NULL Pointer Dereference (CWE-476) only applicable to languages utilizing pointers like C/C++ and Code injection (CWE-94) only applying to interpreted languages. This means that languages other than C/C++ can also be subjected to high-impact vulnerabilities like the Log4Shell vulnerability [27] in Java recently. All of the three aforementioned observations show a dire need for approaches that can perform vulnerability prediction in multiple languages.

Despite the aforementioned benefits, multi-lingual vulnerability prediction poses three key challenges to be addressed. Firstly, the number of programming languages is large, so training and maintaining a separate model for each language as per the current practice is quite resource-intensive and inefficient in practice. A more practical approach is to develop a single model that can consume data in different languages and predict new vulnerabilities in respective languages. Nevertheless, the effectiveness of such a combined model has not been investigated. Secondly, different languages have distinct code syntax, creating potential issues for code representation. An effective representation model needs to capture the nuances in various languages without requiring significant changes to the model architecture. Code models adapted from large pre-trained language models like CodeBERT [17] are advantageous in this scenario because they have the demonstrated ability to capture syntactic and semantic information of code in different languages through masked language modeling [28]. However, the effectiveness of large language/code models for multi-lingual vulnerability prediction has not been well understood. Thirdly, new languages emerge over time, making it expensive to frequently retrain a model from scratch for the new languages. A better way would be to reuse the knowledge of a trained model on existing languages to adapt to a new language. In this case, we only need to train the model on the data of the new language, which would significantly reduce the training time. This process is commonly referred to as incremental learning or continual learning [29]. However, the use of incremental learning to handle new languages for multi-lingual vulnerability prediction is yet to be explored. Overall, to the best of our knowledge, our study is the first to address the above three challenges, aiming to propose an effective and efficient solution to multi-lingual vulnerability prediction.

Incremental learning has become an increasingly important yet under-explored area of research within Software Engineering, particularly for tasks that involve evolving datasets and the need for models to adapt over time without forgetting previous knowledge. This concept is crucial in software engineering due to the dynamic nature of software development and the continuous integration of new code and features.

Pamela et al. [30] presented an innovative approach that combines incremental learning with multi-feature tossing graphs. This method allows for the continuous updating of the bug triage system with new data, improving its accuracy and efficiency over time. By incorporating fine-grained incremental learning, the system can adapt to new patterns in bug reports and developer activities without discarding the valuable knowledge accumulated from historical data. Zi et al. [31] applied incremental learning to the prediction of bugs in source code changes. This approach is particularly relevant in continuous integration and deployment environments, where code changes are frequent and models must rapidly adapt to new data. By employing incremental learning, the model can update its predictions based on the most recent changes, maintaining high accuracy in bug prediction over time. More recently, Jingmei et al. [32] addressed the challenge of classifying malware in scenarios where only limited data is available. This work leverages incremental learning to effectively update the classification model as new malware samples are discovered, ensuring that the model remains current and effective without the need to be retrained from scratch on the entire dataset.

These prior studies have demonstrated the potential of incremental learning in addressing the challenges of software engineering tasks that require continuous adaptation. Our work adds to the body of knowledge by investigating the capability of incremental learning for vulnerability prediction. Specifically, to the best of our knowledge, we are the first to leverage incremental learning to enable a multi-lingual vulnerability prediction model to handle a new language without resource-intensively retraining the model on the data of existing languages.

MVD aims at detecting software vulnerabilities in multiple programming languages simultaneously. Specifically, our model is designed to detect function-level vulnerabilities in different languages. In contrast, as illustrated in Fig. 1, conventional vulnerability detection models are often constrained by their language-specific designs, where each model is typically tailored for a single programming language. This results in the need for separate models for each new language, introducing redundancy and leading to inconsistent vulnerability detection mechanisms across different languages.

To enable multi-lingual vulnerability prediction, MVD capitalizes on the CodeBERT [33] pre-trained language model. CodeBERT was chosen because it has been pre-trained with CodeSearchNet [34], a large code corpus of multiple programming languages, which allows CodeBERT to discern and encapsulate the intricate lexical and logical nuances of diverse code snippets, yielding a detailed vector representation. It is also worth noting that CodeBERT is currently the state-of-the-art model for function-level vulnerability prediction [24]. However, MVD is unique and innovative in two significant ways compared to existing vulnerability prediction models using CodeBERT (e.g., [11, 20]).

Firstly, MVD’s training phase involves processing labeled vulnerability datasets from multiple languages simultaneously, adopting a multi-class classification approach. During training, MVD is trained to differentiate multiple classes corresponding to the existence of vulnerabilities in different languages. Note that non-vulnerable/clean code is a separate class. For inference, these vulnerable classes are consolidated into a single ”vulnerable” category, enabling the model to perform a binary classification task. This approach allows MVD to assimilate shared vulnerability patterns across languages, enhancing its detection performance.

Secondly, the utilization of incremental learning in the MVD framework ensures it can effortlessly accommodate new languages without extensive retraining. This positions MVD as a progressive solution in software vulnerability detection, primed to adapt to the ever-evolving programming language landscape.

We use the WordPiece [33] tokenizer, which is aligned with the CodeBERT pre-trained model employed in our framework. WordPiece is a data-driven tokenization method that iteratively breaks down words into commonly occurring subwords or merges frequent subwords. While it shares similarities with the Byte-Pair Encoding (BPE) [35] method, where frequent pairs of characters are merged into single tokens, there are subtle differences. Specifically, while BPE focuses on merging the most frequent character pairs, WordPiece prioritizes subwords based on the likelihood of their occurrence in the data. This distinction allows WordPiece to be more adaptive in representing rare words. In the context of source code vulnerability detection, WordPiece’s ability to handle out-of-vocabulary words by representing them as a sequence of subwords is invaluable. It ensures that even unique identifiers and terminologies in source code can be meaningfully represented, bolstering the model’s accuracy in vulnerability detection.

Our MVD model inherits the initial weights from the pre-trained CodeBERT [17], which provides a robust foundation for understanding programming languages. Upon receiving source code, the model begins by tokenizing the input using the WordPiece tokenizer (see Section III-B). The tokenized results are then passed through a word embedding layer, which maps each token to a high-dimensional space, capturing the semantic and syntactic nuances of the tokens. Additionally, positional encoding is applied to each token to retain the order information, which is crucial in understanding the structure of the code.

The embedded tokens, now enriched with positional information, are subsequently processed by a stack of 12 transformer layers. These layers, through self-attention mechanisms, enable the model to capture dependencies between tokens regardless of their positions in the input sequence. The output from the final transformer layer corresponding to the <cls> token, which aggregates the contextual information of the entire sequence, is then fed into a multi-class linear classification layer.

The classification layer produces logits, which are essentially raw predictions that have not yet been normalized. These logits are then transformed into probabilities using the softmax function, which assigns a probability to each class, indicating the likelihood of the input code belonging to a particular vulnerability class or being clean.

During the training phase, as illustrated in the bottom left of Fig. 1, the output dimension is $n+1$, accounting for a ‘clean’ class and $n$ vulnerable classes corresponding to the $n$ programming languages. This design mirrors our training data, which comprises labeled vulnerable functions from $n$ languages and their non-vulnerable counterparts. We employ the gradient descent algorithm to minimize the FOLA loss function, a variant designed to address the class imbalance issue, which we describe in Section III-D. The model iteration with the highest performance on the validation set is selected as the final trained model.

In the testing phase, the model’s output dimension is binary, distinguishing between ‘clean’ and ‘vulnerable’ classes. Here, the specific language of the vulnerability is not of interest; rather, the focus is on the binary determination of the presence of a vulnerability. To achieve this, the probabilities for the $n$ different vulnerable classes are summed up, resulting in a single probability representing the overall likelihood of the code being vulnerable. This aggregated probability is then compared to the probability of the ‘clean’ class to make the final binary decision.

In the domain of multi-class vulnerability detection, the class imbalance phenomenon presents a significant challenge [36]. Typically, the datasets used for training such models have a disproportionate number of examples across different classes. This imbalance often results in a model that is biased toward the majority class, leading to suboptimal performance on the minority classes, which are usually the more critical vulnerabilities to detect. We did not use sampling techniques to tackle class imbalance in our study as these techniques lose information in training data (i.e., under-sampling) and/or increase training size and time too significantly (i.e., over-sampling).

To mitigate this issue, we adopt a hybrid approach that combines the principles of Focal loss [37] with the logit adjustment method [38]. The Focal loss function is designed to focus more on the hard-to-classify examples by reducing the relative loss for well-classified examples, thus putting more emphasis on correcting the misclassified examples. The logit adjustment method, on the other hand, aims to recalibrate the logits of each class based on their frequency, effectively adjusting the decision boundary for each class. The combined loss function, which we refer to as FOLA loss, is formulated as follows,

where $p_{t}$ is the model’s estimated probability for the true class $t$, $\alpha_{t}$ is a weighting factor to balance the importance of different classes, and $\gamma$ is the focusing parameter of the FOCAL loss that effectively reduces the loss contribution from easy examples and increases the importance of correcting misclassified examples. The term $\tau\log(q_{t})$ represents the logit adjustment for class $t$, where $\tau$ is a hyperparameter that controls the strength of the adjustment and $q_{t}$ is the frequency of class $t$.

By applying this FOLA loss function during the training of our MVD model, we can effectively address the class imbalance by dynamically adjusting the contribution of each class to the loss based on its frequency and the difficulty of classifying its examples. This ensures that the model does not become biased toward the majority class and improves its performance on the minority classes, which is crucial for achieving reliable performance of vulnerability detection across multiple programming languages.

In the realm of software development, the sheer number of programming languages presents a daunting challenge for vulnerability detection models. It is impractical to encompass all existing languages in the initial training phase of a model. In real-world applications, the deployed MVD model may encounter projects written in languages not included in its training corpus. Compounding this issue is the frequent scenario where users lack access to the original training data [39], making it difficult to leverage past knowledge.

A naive solution might involve training a separate model for each new language from scratch, but this approach is fraught with inefficiencies, leading to model redundancy and a failure to capitalize on the knowledge embedded within the already trained MVD model.

To circumvent these issues, our MVD framework incorporates an incremental learning [40] module. This module enables the model to extend its capabilities to new languages by building upon the knowledge acquired during its initial training. This method not only facilitates the learning of new tasks by leveraging existing capabilities but also largely maintains the model’s performance in the original languages.

As shown in the Fig. 1, we implement this module by introducing a distillation loss that retains the knowledge of the original languages while accommodating new information. The equation for the distillation loss is as follows,

where $z_{i}$ represents the output logits of the new model for the original languages, $\sigma$ denotes the softmax function, and $z_{i}^{old}$ are the logits from the previously trained model. This loss ensures that the predictions for the original languages remain consistent before and after the model is updated.

The final loss function is a composite of the distillation loss and the FOLA loss, which addresses the class imbalance issue:

By optimizing this combined loss function during the training process, MVD can effectively learn to detect vulnerabilities in new programming languages while preserving its existing knowledge base. This approach streamlines the extension of the model’s capabilities and ensures that the learned information is retained and utilized effectively.

We set out to answer the following three Research Questions (RQs) to shed light on the effectiveness of MVD for multi-lingual software vulnerability detection.

• •

  RQ1: Can MVD outperform state-of-the-art models for software vulnerability detection in different languages?

• •

  RQ2: What are the contributions of the key components in MVD to the model performance?

• •

  RQ3: What is the performance of MVD when extended to a new language?

RQ1 seeks to evaluate the effectiveness of MVD against current leading models in the field of software vulnerability detection across various programming languages. Given MVD’s architecture, which leverages the pre-training on data of multi-lingual vulnerabilities and a novel class-imbalance loss function, it is hypothesized that MVD can provide superior performance by effectively learning from a diverse set of vulnerability patterns across multiple languages. RQ2 aims to dissect the MVD framework to understand the impact of its individual components on the overall model performance. This involves analyzing the role of the multi-class classification paradigm, the FOLA loss function, and the strategy of fine-tuning either the entire model or just the classifier layer. By conducting an ablation study, we can determine how each component contributes to the model’s ability to detect vulnerabilities and whether they are all critical to achieving the observed performance levels. RQ3 addresses the model’s adaptability and performance when extended to a new programming language not included in the initial training data. This question is crucial for understanding the practicality of MVD in real-world scenarios where it may need to be applied to languages that emerge or become relevant after the model has been deployed. RQ3 is expected to shed light on the extent to which the incremental learning module can integrate new languages without significant loss of performance on previously learned languages.

We customized the methods and tools provided by CVEfixes [41] to curate vulnerability data for six different programming languages, namely C/C++, Python, Java, C#, JavaScript, and TypeScript. C and C++ were chosen because they have been commonly investigated in the literature (e.g., [9, 8, 11, 20]). The other five are the most popular languages in practice, according to the developers’ survey conducted by Stack Overflow.³³3https://survey.stackoverflow.co/2023/#most-popular-technologies-language-prof Note that we focused on general-purpose programming languages, so we excluded task-specific languages like SQL for database manipulations, HTML/CSS for web development, or Bash for scripting on Linux-based operating systems. We first collected vulnerability-fixing commits in each of the aforementioned languages reported on the National Vulnerability Database [42]. In these commits, the functions encompassing lines changed were considered vulnerable; otherwise, they were non-vulnerable. Note that this data curation process follows the same practice of Big-Vul [43], the largest vulnerability dataset widely used in the literature.

To further increase the data quality, we applied a series of filtering steps to the collected functions. To ensure that a function was written in a particular language, we defined the file extensions for each language, as given in Table I. Note that these extensions might not cover all available (vulnerable) code of each language in the wild, but they are the most commonly used ones in practice, ensuring the majority of code was curated. We also removed the functions inside test files to focus on production code. We also discarded functions that contained only cosmetic (non-functional) changes, e.g., changing whitespaces/newlines/comments as these functions were unlikely to contain vulnerabilities. These filtering steps are common practices in the literature (e.g., [44, 45, 23]). We did not trace/include latent vulnerable functions as there is not yet an accurate way to automatically determine the origin (introduction time) of vulnerabilities [45]. After the filtering steps, the number of vulnerable and non-vulnerable functions are reported in Table I. It is evident that the number of vulnerable functions was significantly smaller than that of non-vulnerable ones, confirming our argument about the existence of class imbalance in multi-lingual vulnerability prediction.

In our experiments, we assessed the MVD framework using a variety of evaluation metrics that are widely accepted in vulnerability detection research. We incorporated the Area Under the Precision-Recall Curve (PR-AUC) as a key metric, which aggregates the precision-recall curve into a single value. PR-AUC is particularly advantageous in our setting as it evaluates the model’s performance across all thresholds (threshold-agnostic), offering a measure that is unaffected by the selection of any specific decision boundary. This metric is also crucial for imbalanced classification like multi-lingual vulnerability detection (see Table I), where the cost of missing a true vulnerability (low recall) and the expense of investigating a false alarm (low precision) must be carefully balanced. By using PR-AUC, we gain insight into the model’s ability to discern between vulnerable and non-vulnerable code snippets across the entire spectrum of precision and recall, providing a robust indicator of its overall predictive quality.

We also employed the F1-score of the binary classification for its balanced consideration of precision and recall, making it particularly relevant for our imbalanced dataset where true negatives vastly outnumbered true positives. Precision is critical to ensure the model minimizes false positives, which can be costly and time-consuming in practical applications, while recall is essential for capturing as many true vulnerabilities as possible to maintain system security. The Matthews correlation coefficient (MCC) was also used due to its effectiveness in providing a nuanced view of the model’s performance across all quadrants of the confusion matrix, which is valuable in our context of imbalanced classes.

In this research question, we aim to compare the performance of a model trained on a multi-lingual dataset encompassing all six languages against models trained exclusively on single-language datasets. For each language, we partitioned our dataset into training, validation, and testing sets following an 8:1:1 ratio, which has been the standard for vulnerability prediction (e.g., [23, 11, 20, 24]). The model was trained on the training set, and its performance was assessed on the validation set after each epoch. The iteration achieving the highest PR-AUC on the validation set was preserved as the final model and tested on the testing set.

Regarding hyperparameters, we set the initial learning rate to $2\times 10^{-5}$ and employed a cosine annealing schedule, gradually reducing the learning rate in a cosine curve-like fashion as training progresses. This approach helps in fine-tuning the learning rate to converge optimally. We utilized the backpropagation algorithm and the AdamW optimizer [46], a variant of the Adam optimizer [47] that is particularly effective for fine-tuning Transformer-based models. This optimizer updates the model weights to minimize the loss function.

Upon completion of the training phase, we evaluated the model’s performance using the testing set to ensure an unbiased assessment of its generalization capabilities. For baseline comparisons, we adopted LineVul [11], the state-of-the-art vulnerability prediction model [24], which involves fine-tuning CodeBERT using single-language vulnerability datasets. Consequently, we obtained distinct models for each language: LineVul-Python, LineVul-C/C++, LineVul-Java, LineVul-C#, LineVul-JavaScript, and LineVul-TypeScript. We utilized the source code from LineVul⁴⁴4https://github.com/awsm-research/LineVul and retrained the models using our datasets. The data split ratio, hyperparameters, model selection criteria, and evaluation procedures were consistent with those used for the multi-lingual model to ensure a fair comparison.

To ascertain the individual impact of MVD’s components, we conducted an ablation study using the same evaluation setup as in RQ1. We systematically removed or altered certain components to observe the change in performance, thereby validating the significance of each component.

Firstly, we compared the full MVD model, which employs a multi-class classification paradigm, with a variant we term MVD-binary. The MVD-binary model simplifies the problem by aggregating all vulnerable examples into a single class, regardless of the language. This binary classification approach aligns with traditional single-language vulnerability detection models, where the classifier is binary. By comparing the performance of MVD-binary with the full MVD model, we can assess the efficacy of the multi-class approach in enhancing the model’s discriminative power across multiple languages.

Next, we turned our attention to the FOLA loss function, which is a composite of Focal loss and logit adjustment. To evaluate its effectiveness, we trained variants of the MVD model using different loss functions: one with Focal loss alone, one with cross-entropy combined with logit adjustment, and one with the standard cross-entropy loss. By comparing these variants, we can determine the contribution of the FOLA loss function to the model’s ability to handle class imbalance and improve performance for minority classes.

Lastly, we explored the utility of using the base CodeBERT model solely as a feature extractor, wherein its weights remain frozen during the training of the classifier. This approach can preserve the original representations learned by CodeBERT and expedite the training process. By comparing this method with the full model training, where CodeBERT’s weights are fine-tuned during training, we can discern whether the additional fine-tuning step significantly contributes to the model’s performance or if the pre-trained representations are sufficient for vulnerability detection tasks.

Through this ablation study, we aim to shed light on the necessity and efficiency of each component and training strategy within the MVD framework, providing insights into their roles in achieving the model’s overall performance.

Our experiment was designed to investigate the adaptability of the MVD model when it is extended to accommodate a new programming language. This process was conducted in two distinct stages to simulate the scenario where a previously unencountered language needs to be integrated into an existing model. The data splits were the same as in RQ1.

Initially, we prepared the groundwork by training six separate MVD models, each intentionally omitting one of the languages from the training data. This language, excluded in the first stage, was designated as the ‘new’ language for the subsequent phase of the experiment. By doing so, we created a baseline for how the model performs without any prior knowledge of the new language.

In the second stage, we employed the incremental learning module, described in Section III-E, to introduce the new language to the pre-trained MVD models. This step allows us to observe how the model assimilates new information and whether it can leverage the knowledge acquired from the original languages to enhance its performance on the new language.

Upon completion of the incremental learning process, we conducted a series of comparisons to evaluate the efficacy of this approach. We measured the performance of the MVD model on the new language and compared it with that of a single-language vulnerability detection model trained solely on the new language. This comparison aims to highlight the advantages of using a multi-lingual model that can transfer learned knowledge to new contexts/languages.

Furthermore, we assessed the performance of the original languages both before and after the application of incremental learning. This comparison is essential to ensure that the extension process does not detrimentally affect the model’s existing capabilities.

Finally, we compared the performance of the incrementally updated model with the MVD model that was trained with all six languages from the outset. This comparison is intended to illustrate the gap, if any, between the incrementally learned model and the theoretical optimum, where the model has been trained on all languages simultaneously.

Overall, RQ3 aims to not only validate the incremental learning approach but also to quantify its impact on both the new and original languages, thereby providing a better understanding of the model’s extensibility and robustness in the face of evolving software development practices.

Table II presents a comparative analysis between the performance of the multi-lingual Vulnerability Detection (MVD) model and the single-language LineVul models based on CodeBERT, which were trained on individual programming languages. Each sub-table is titled with the language used for testing, and the rows labeled LineVul-language represent the LineVul models trained specifically for that language. The colors red and blue in the table highlight the top-1 and top-2 performance metrics, respectively.

The experimental results showcased that the MVD model consistently achieved top-tier performance, either ranking first or at least second, often outperforming the single-language LineVul models trained on their respective languages. The significant improvements included 34.9% for C/C++, 30.7% for Java, and 24.9% for TypeScript in terms of PR-AUC. For the remaining languages, MVD performed on par (within 5% in PR-AUC) compared to that of the single-language LineVul counterparts. The general trend of the MVD model outperforming the baselines was also observed for the other metrics. These results confirm the effectiveness of our unified MVD model, confirming that training across multiple languages can leverage cross-linguistic knowledge of vulnerabilities and significantly improve vulnerability detection efficacy.

Overall, the results illustrated that a single MVD model could effectively operate across different languages, unlike the LineVul models, which often exhibited substantial performance declines when tested outside their training language. On average, MVD had 83.7%, 167.2%, 137.9%, 101.5%, 125.6%, and 193.6% better performance (PR-AUC) of predicting vulnerabilities in all six languages than the state-of-the-art LineVul models trained specifically for Python, C/C++, Java, and JavaScript, C# and TypeScript, respectively. It is also worth noting that MVD was approximately 7% better PR-AUC than that (0.5008) of the LineVul models trained for each language individually and requiring nearly five times more resources. All these results highlight the MVD model’s superior capability to identify vulnerabilities in software projects developed in multiple languages, thereby enhancing its practical utility in diverse development environments.

The impacts of the components on the performance of our MVD model are shown in Table III. The different variants of the MVD model included in the ablation study are described and analyzed hereafter.

Firstly, we compared the full MVD model, which employs a multi-class classification paradigm, with a variant termed MVD-binary. The MVD-binary model simplifies the problem by aggregating all vulnerable examples into a single class, irrespective of the language. This binary classification approach aligns with traditional single-language vulnerability detection models, where the classifier is binary. While the MVD-binary model performed competitively, the full MVD model, utilizing a multi-class approach, outperformed the MVD-binary model in all metrics across all languages (only except Recall in JavaScript), as well as by 4% in PR-AUC, on average. This highlights the improved efficacy of the multi-class approach over the conventional binary counterpart.

We next assessed the efficacy of the FOLA loss function, a hybrid of Focal loss and logit adjustment. Variants of the MVD model were trained using distinct loss functions: MVD-focal, employing Focal loss; MVD-lace, combining cross-entropy with logit adjustment; and MVD-ce, using standard cross-entropy. The outcomes demonstrated that the FOLA loss function was superior in managing class imbalance and enhancing performance in minority classes, consistently achieving at least top-2 PR-AUC across all languages. Conversely, models employing other loss functions exhibited performance variability across different languages, complicating the task of achieving balanced performance. Regarding the average performance, the MVD outperformed the models with all other loss functions by at least 0.2% in PR-AUC. These results substantiate the beneficial impact of the FOLA loss function.

Lastly, we explored the utility of using the base CodeBERT model solely as a feature extractor (MVD-freeze), with its weights remaining frozen during the training of the classifier. This approach aimed to preserve the original representations learned by CodeBERT and expedite the training process. However, the results indicated that MVD-freeze struggled to achieve comparable performance to the fully fine-tuned MVD. Specifically, the average performance showed a significant gap of around 30% in PR-AUC. These findings suggest that fine-tuning CodeBERT during training significantly enhances the model’s performance.

All the above observations elucidate the necessity and efficiency of every component and training strategy within the MVD framework. The results have provided insights into the roles of multi-class classification, the FOLA loss function, and fine-tuning of pre-trained models in achieving superior performance in multi-lingual vulnerability detection, confirming the overall effectiveness of our MVD framework.

This experiment was conducted to assess the adaptability of the MVD model when a new programming language is integrated incrementally and the training data of old language(s) is no longer accessible. The results in Table IV highlighted several promises of the effectiveness of incremental learning compared to training a model from scratch with all languages.

The incremental learning approach (inc-X, where X represents the newly introduced language) was generally better for four out of six languages in PR-AUC for the new language compared to models that were trained on that language alone (LineVul). This trend indicates that the incremental learning approach can effectively assimilate new information and improve the model’s performance in the newly added language.

The results also revealed that the performance on the original languages did not degrade significantly and could even improve following the incremental learning process. For example, when C/C++ was incrementally added, the performance of the model on Python decreased from 0.9018 (w/o-C/C++) to 0.8846 (inc-C/C++), which suggests some loss. However, we also witnessed performance increase after incremental learning; for instance, when Java was incrementally added, the performance on TypeScript even improved (from 0.1653 to 0.2860). This suggests that the model retains much of its original knowledge even when the training data of old languages are unavailable. Further, we observed that the performance of MVD for each of the six languages after incrementally extending to a new language could vary. For example, the PR-AUC of MVD for TS ranged from 0.1260 to 0.2860 when incrementally learned in different languages. In addition, incrementally training MVD in a new language did not always lead to better performance for that language than incremental training in other languages. These results imply that the language-wise performance of MVD after incremental learning depends on the combination and order of the languages on which the model was previously trained.

Furthermore, we compared the incrementally trained models (denoted as inc-X) with the MVD model trained on all six languages. While the incrementally trained models could occasionally outperform the full MVD model on specific languages (e.g., inc-Python slightly outperforming the MVD model for Python by 0.5% in PR-AUC), the full MVD model generally achieved higher PR-AUC across all languages. This suggests that the MVD model benefits from simultaneous multi-lingual training, capturing diverse and shared patterns and features that lead to a more stable and balanced performance overall. However, incremental learning remains an effective strategy for efficiently adapting the model to new languages without retraining from scratch, despite not always reaching the peak performance of a fully trained multi-lingual model using the data of all available languages.

Overall, the experimental results have demonstrated the efficacy of the incremental learning approach in extending the MVD model to accommodate new languages while largely preserving its performance on previously known languages. Although the incrementally trained models do not achieve the same level of performance as a model trained with all languages from the start, they still provide a viable solution for environments where retraining on all data is impractical.

The threats to construct validity concern the data selection in multiple programming languages. We utilized the methods and tools provided by CVEfixes, one of the largest multi-lingual vulnerability datasets in the literature, for our data collection. This dataset follows the latest practice to curate vulnerable and non-vulnerable functions. On top of the data provided by CVEfixes, we also improved the quality by removing the code irrelevant to vulnerability based on the recent checklist of vulnerability data quality assessment [48].

The internal validity threats are related to the optimality of the vulnerability prediction models. With limited computational resources, we could not try all possible hyperparameters. We still tuned our models using the common hyperparameters from relevant studies. For the LineVul baseline model, we also leveraged the recommended hyperparameters to tune it. The performance of our proposed MVD model may not be the highest possible for multi-lingual vulnerability prediction, but at least it established a strong foundation for future work to compare with and build upon.

The external validity threats are pertinent to the generalizability of our findings. We performed experiments and analysis of MVD using data from hundreds of projects in six different languages and various application domains, but our findings may still not generalize to other languages.