Active Code Learning: Benchmarking Sample-Efficient Training of Code Models

Active learning is a well-known technique to enable sample-efficient model training. Figure 1 depicts the overview workflow of active learning. Given an unlabeled dataset and a model under training, the first step of active learning is to choose the features used for conducting data sampling. Generally, two types of features can be used, 1) the data features itself (e.g., image pixels, and code tokens), and 2) features extracted from the model (e.g., output probabilities, and code embeddings). After obtaining the features, acquisition functions are used to select the most valuable data for labeling. Here, important means the model can learn more information from this kind of data and achieve better performance on test data. Finally, developers label these selected data and use them to train the model. In this way, developers can train a model under a fixed labeling budget. When a new budget is allowed, developers repeat this process and further enhance the trained model.

The acquisition function is the most important part of active learning which decides the quality of labeled training data. Existing acquisition functions can be roughly divided into two groups, output-uncertainty-based functions, and clustering-based functions. Output-uncertainty-based functions obtain the model predictions of the unlabeled data first and then use some uncertainty metrics (Entropy of output probabilities) to rank these data and select the most uncertain ones for training. The intuition behind this type of function is that uncertain data are usually close to the decision boundaries of the model, thus, training the model using these data can force the model to build clear boundaries and improve its correctness. Clustering-based functions assume that the models have better performance if they learn diverse more data information. Thus, people use different distance methods to measure the distance of data to each other and select the central data for model training, e.g., using K-means to do data clustering and then select the center data.

In this work, we follow the previous works (Hu et al., 2021; Weiss and Tonella, 2022) and collect and implement 11 acquisition functions for active code learning. Let $X$ be the training data and $x\in X$ be an input sample. $N$ is the number of classes. $y$ and $y_{x}$ are the ground-true label and predicted labels. $p_{i}(x)$ represents the predicted probability of $x$ belonging to the $i$th class. Table 1 presents these acquisition functions with their brief descriptions. For detailed information on each function, please refer to the original paper.

ML4Code (Allamanis et al., 2018) is a new but hot direction in the current software engineering research field. The basic idea of ML4Code is to train a machine learning model to solve software problems, for example, program repair, vulnerability detection, and code summarization. Machine learning models and training data play key roles in ML4Code. For machine learning models, due to the naturalness of software, researchers believe model architectures that are good at handling text data are also promising for solving code data. Thus, the famous code models mainly come from modifying natural language models, e.g., CodeBERT (Feng et al., 2020a). The training data are usually processed by some code representation techniques to transfer the raw data to a machine-readable format, e.g., transfer the strings of code to a sequence of tokens with integer format. How to design this transformation (i.e., code representation) is another research topic that highly affects the performance of trained models.

In this work, we focus on the most practical paradigm of ML4Code – building models of code for specific downstream code tasks by fine-tuning pre-trained code models. Existing studies (Feng et al., 2020a; Guo et al., 2020) already demonstrated that fine-tuning pre-trained models can achieve better results than training the models from scratch. Here, the pre-trained code model has been trained on multi-language datasets, therefore, they already learned general information of code, e.g., the pre-trained CodeBERT model learns knowledge from six datasets with different programming languages. As a result, developers only need to prepare their dataset for a specific task and use them to fine-tune the pre-trained model.

To address the question of suitable feature selection (RQ1), we first prepare different versions of acquisition functions based on the features they used. Then, we conduct active learning on different code tasks using these functions and compare the performance of trained models. we can draw conclusions about which features are most suitable for each acquisition function. Here, we focus on three types of features from different perspectives, 1) code embeddings, 2) sequence of code tokens, and 3)model outputs vectors.

• •

  Code Embeddings It is common to consider code embeddings as the input of clustering methods since code embeddings produced by pre-trained code models can present the general information of code. Code embedding is similar to the image data after image pre-processing. In our study, code embeddings extracted from the fine-tuned code models which can better represent the downstream task are used as the features.

• •

  Sequence of code tokens Regardless of the code representation techniques, the raw program will be converted into a sequence of integer values. Thus, it is also possible to use these sequences as the inputs of the clustering methods. It is similar to using the pixel numbers of images as inputs. The only difference is that code tokens are often with bigger data space, i.e., the image pixels are from 0 to 255, while the range of code tokens depends on the vocabulary size, which is generally much bigger than 255, e.g., the vocabulary size of our used CodeBERT model is 50265.

• •

  Model outputs Different from the above two features that have been widely explored in other fields and can be easily considered for active code learning, in our study, we propose to use the model outputs as the input features of clustering methods. The intuition behind this idea is that the output can be seen as the understanding of the model on this input data which should be useful for data selection. Specifically, for classification tasks, we use the one-hot output probabilities as the features, and for the non-classification tasks, the output vectors produced by decoders are used as the features.

For the comparison of each acquisition function  (RQ2), the main goal is to find the recommended function that has consistently better performance than others in active code learning. At the same time, we also compare the output-uncertainty-based functions and clustering-based functions to determine if the previous findings (Weiss and Tonella, 2022) hold true in active code learning. Here, based on the first study, we use suitable features as the input for clustering-based acquisition functions and perform code selection.

Table 2 presents the datasets and models we used in the study. We consider three code tasks including a multi-class classification task (problem classification), a binary classification task (clone detection), and a non-classification task  (code summarization).

• •

  Problem classification is a multi-class classification task. Given a program, the model predicts the target problem that the program solves. We use the dataset JAVA250 (Puri et al., 2021) provided by IBM for this task. JAVA250 contains 250 code problems, e.g., problem: write a program which prints heights of the top three mountains in descending order.

• •

  Clone detection is a well-studied task in the software engineering field to lighten the effort of software maintenance. The main purpose of this task is to check if two programs are semantically equivalent. Thus, code clone detection is often seen as a binary classification problem. In our study, we use the dataset provided by BigCloneBench (Svajlenko et al., 2014) which contains a large number of Java code clone pairs. Besides, for the computation friendly, we follow the previous work (Yang et al., 2022) and only use a subset of data from BigCloneBench.

• •

  Code summarization is a common code task for helping developers understand code snippets. Given a program, the model generates natural language comments to describe the functionality of this program. We use two datasets provided by Microsoft (Lu et al., 2021) in our study.

For the code models, we focus on pre-trained code models since they achieved much better results than models trained from scratch (Feng et al., 2020a; Guo et al., 2020). We follow the previous work (Yang et al., 2022) and use two well-known pre-trained code models in our study, CodeBERT (Feng et al., 2020a) and GraphCodeBERT (Guo et al., 2020). Other models can be easily added and evaluated to our provided project. Considering the downstream tasks, the pre-trained model is used as an encoder to generate code embeddings, and a decoder is followed to produce the final results for a specific code problem. For example, a dense layer is used as the decoder to produce the output probabilities of classification tasks. The same as the original works or CodeBERT and GraphCodeBERT, during the fine-tuning, we also fine-tune the parameters of pre-trained encoders to ensure a better performance on downstream tasks.

• •

  CodeBERT is a bimodal model that shares a similar architecture with the well-known BERT model in the field of natural language processing. The CodeBERT model is initialized through pre-training with six code datasets featuring various programming languages such as Java and Python. During the training process, programs are transformed into sequences of tokens, which is similar to text data. As a result, CodeBERT is able to learn the semantic information of code data.

• •

  GraphCodeBERT is a newer pre-trained code model that incorporates both sequences of tokens and data-flow information to learn code knowledge. This allows pre-trained code models to understand the structural information of programs and generate more precise code representations. In our study, the base model is learned by this combination of code information and fine-tuned for our considered downstream tasks only using the code token information. The reason is that we found in some downstream tasks, adding the data-flow information to fine-tune the model can bring a negative influence on the performance of models, e.g., for the problem classification task, adding data-flow information and without data-flow information, the accuracy of fine-tuned models are 82.30% and 98.39%, respectively.

Our study contains three types of code tasks. For each task, we use the most practical metrics to evaluate the trained models.

Accuracy on the test data is the basic way to evaluate the performance of multi-class classification models. It calculates the percentage (%) of correctly classified data over the entire input data.

F1-score is a commonly used metric for binary classification problems. It calculates the harmonic mean of the precision and recall scores. Given that, true positive (TP) represents the number of samples correctly predicted as positive, false positive (FP) represents the number of samples wrongly predicted as positive, and false negative (FN) represents the number of samples wrongly predicted as negative, F1-score is calculated as:

Perplexity (PPL) is a widely used metric to evaluate language models. PPL can be seen as the loss of language models that can record the logs of the training process. A lower PPL score means a better performance of the model. Recently, researchers applied PPL to record to evaluate the code summarization models (Wan et al., 2018). Specifically, PPL is calculated by:

BLEU (Bilingual Evaluation Understudy) is a metric to evaluate the quality of the generated text to another (the reference). Simply, BLEU score is calculated by:

The value of N depends on the used N-gram precision, and the weights $w_{n}$ = N / 4. in our study, 4 is used. $p_{n}$ is the ratio of length n sub-sequences in both the candidate sequence and the reference. BP is the brevity penalty calculated by:

Training configuration. Considering the hyperparameters used for model training, we follow the previous work (Yang et al., 2022) and use the same batch size and learning rate for each model. All the detailed settings can be found on our project site. For model fine-tuning, we set the training epoch as 10 which is enough to ensure the convergence of models.

Active learning configuration.We initialize the models (which is a common setting of active learning that we start from a model with a little knowledge of the datasets) by training them on randomly selected 500 samples. We set the labeling budget as 1% of the entire training set and do 10 times iterations of active learning.

Acquisition function configuration. For all clustering-based methods, we follow the previous work (Hu et al., 2021) and set the number of centers as the labeling budget. For BALD, we set the times of dropout prediction as 20.

The project is based on Python-3.6 and PyTorch-1.10.2 framework. The key implementation of code models is modified from the open source project  (Lu et al., 2021). We adopt acquisition functions implemented by (Hu et al., 2021; Weiss and Tonella, 2022) that are used for image data and text data to code data. All the source code can be found on our project site. We conduct all the experiments on a 2.6 GHz Intel Xeon Gold 6132 CPU with an NVIDIA Tesla V100 16G SXM2 GPU. We repeat all the experiments five times to reduce the influence of randomness and report the average results in the following sections.

First, we explore the features of clustering-based acquisition functions. Figure 2 depicts the performance of models trained by different labeling budgets. Here, we only list the results of CodeBERT models and the whole results can be found on our project site.

From the results, we can see that the same acquisition function could train models with significantly different performance depending on the features used. In particular, for the K-Means, K-Center, and Coreset functions, the performance difference is substantial and should not be overlooked. In contrast, the BADGE function is relatively stable compared to the others. Then, we analyze the most suitable features that should be used for each function for solving each code task. Table 3 displays the percentage of each function that produces models with the best performance across all labeling budgets. Surprisingly, overall, the results demonstrate that the output vectors extracted from the models are superior features compared to code tokens and code embeddings. This suggests that active code learning should focus on output vectors rather than input vectors when conducting data selection, unlike active learning for image and text data.

Besides, we observe that the feature selection of some acquisition functions also depends on the types of downstream tasks. In both classification tasks, output vectors and code embeddings are the best choices for K-Means and K-center, respectively. However, In the non-classification code summarization task, the choices become code tokens and output vectors. On the other hand, output vectors are the most useful features for BADGE and Coreset regardless of the type of tasks.

Based on the experimental results, we provide recommendations for the feature selection in clustering-based acquisition functions for active code learning.

• •

  K-Means-C (KM-C): use model output vectors (code tokens) for classification (non-classification) tasks.

• •

  K-Center-C (kC-C): use code embeddings (model output vectors) for classification (non-classification) tasks.

• •

  BADGE-C: use model output vectors for all code tasks.

• •

  Coreset-C: use model output vectors for all code tasks.

After studying the feature selection, we compare all acquisition functions on different code tasks. Here, we use our recommended features to build the clustering-based function. Table 4 presents the results of classification tasks.

For the multi-class classification task (problem classification), we can see that output-uncertainty-based methods often achieve better results than the clustering-based methods. In which, Margin which only uses the top-1 and top-2 probabilities of the outputs perform the best in 5 out of 6 cases. This phenomenon draws a similar conclusion to the previous work (Weiss and Tonella, 2022), that simple methods perform well on active learning. However, considering the binary classification task (clone detection), interestingly, the results show that the previous conclusion can not stand. In all cases, clustering-based methods outperform simple methods (output-uncertainty-based methods). Thus, the first conclusion we can draw is – no acquisition functions that consistently perform better than others, and findings (Weiss and Tonella, 2022) from previous works can not be directly applied to active code learning.

Then, move to the non-classification task (two code summarization tasks), table 5 and table 6 show the results. The first finding is that we will get different conclusions by using different evaluation metrics. For example, the PPL scores demonstrate that KC-C is the best acquisition function while the BLEU scores suggest Coreset-C. However, regardless of the evaluation metrics we used, the gap between the performance of active learning-trained models and the performance of models trained by using the entire data is big. For example, for JavaScript-CodeBERT, under 10% labeling budget, the best PPL score and BLEU score we get are 5.1313 and 10.09, which are 33.28% and 29.64% lower than the 3.85 and 14.34 computed from the model trained by entire training data. These results are totally different from the ones drawn by the classification tasks that using 10% data can train a model with similar and even better performance, e.g., for the clone detection task, models trained with 10%  (97.79%) data have better performance than the models trained by entire training data  (97.15%). Therefore, we can say that active code learning in non-classification code tasks is still in a very early stage, the conclusions from classification tasks can not be migrated to non-classification tasks.

To validate our conjectures, we empirically explore the following two problems:

• •

  Is there a correction between the distance of selected data to each other and the performance of trained models based on these data? Through this study, we explore the probability of improving active code learning from the perspective of providing new distance methods for clustering-based acquisition functions.

• •

  Is there a connection between the existing distance methods and the code evaluation metrics? Through this study, we explore whether our proposed distance methods are different from the old ones or not. The positive answer can demonstrate that evaluation-metric-based methods cannot be replaced by the existing distance methods.

Addressing the first problem follows the following steps:

Step 1 We prepare two groups of models, the first group contains the initialized models (the same as the initialized models used in section 3) that represent models in the early stage of active learning, and the second group includes models that have already been trained using 5% of training data that represent models at a late stage of active learning. In this way, we can see if the correlation we want to study holds in models with different performances. Note that we have prior knowledge of the data used to train the models.

Step 2 We conduct active code learning by using Random acquisition function for each group of models 100 times and record the 100 groups of selected data as well as the trained models for further analysis.

Step 3 We measure the accuracy of the 100 trained models on the test data and calculate the average distance between the selected data samples in each group. Finally, we can get 100 accuracy values and corresponding 100 distance values. Here, the distance can be calculated by different methods.

Step 4 We use Spearman’s rank correlation coefficient to compute the correlation between the accuracy and the distance provided by step 3.

For the second problem, we use different distance calculation methods in Step 3 to obtain the distance scores and then use Spearman’s rank correlation coefficient to compute the correlation between these distance methods.

We select one classification task (problem Classification), and one non-classification task (Code summarization for JavaScript programs) to conduct this exploratory study. For both tasks, we choose CodeBERT as our base model. For the distance calculation methods, we consider four in this study, cosine similarity as distance, Euclidean distance, BLEU score as distance, and CodeBERTScore (Zhou et al., 2023) as distance. Here, CodeBERTScore is a state-of-the-art evaluation metric for code generation. It uses pre-trained contextual embeddings to vectorize each token in the reference program and the generated program first. Then, it computes the pairwise cosine similarity between every embedded token in the reference and every encoded token in the generated code. Finally, the maximum similarity score in each row of the pairwise matrix is used to compute the final similarity of these two programs. Since the BLEU score and CodeBERTScore are computed based on the input level, we also compute the cosine distance and Euclidean distance based on the input embedding (code embedding) for a more fair analysis.

Table 7 presents the results of the correlation between data distance and model accuracy. Surprisingly, the results indicate that regardless of the code tasks, when the model has a poor performance, i.e., at the early stage of active code learning, the trained model performance is not related to the diversity (the distance of data to each other) of used training data. However, for a model that was already trained on a few data and with a good performance, i.e., at the late stage of active code learning, the conclusion changed. Considering the classification task, there is a weak correlation between the cosine and Euclidean distance with the accuracy of the models. That means clustering-based acquisition functions that use these two distance calculation methods are promising to train a model with high accuracy. As already shown in table 4, all these acquisition functions based on Euclidean distance achieve good performance in classification tasks. Considering the non-classification task, the results show this correlation becomes weaker, e.g., for cosine similarity, the correlation changes from -0.2555 to 0.1328 in $Model_{l}$. This is also the reason that the existing acquisition functions do not work well on code summarization tasks and have no advantage over random selection as shown in table 5 and table 6. However, on the other hand, we can see there is a weak correlation between the evaluation scores-based distance and the accuracy of models which does not happen in our considered two classification tasks. The correlation result of CodeBERTScore on $Model_{l}$ is significant (with a p-value less than 0.05). These results lead to a promising direction of proposing new acquisition functions that use evaluation metrics as the distance calculation method for the code summarization task.

Table 8 presents the results of the correlation between different distance calculation methods. For models with low performance ($Model_{e}$), there is always a correlation between cosine similarity or euclidean distance with CodeBERSCore, which means CodeBERSCore is able to produce similar distance ranking of data to the existing distance methods in these models. Combining the conclusion from the last study, we can see that for $Model_{e}$, all methods have a similar distance ranking of data, but this ranking is not connected to the performance of models. However, for $Model_{l}$, we can see there is no correlation between cosine and euclidean to the BLEU and CodeBERTScore. That is the reason why cosine and euclidean (BLEU and CodeBERTScore) correlate with the model performance while BLEU and CodeBERTScore (cosine and euclidean) do not have this correlation in our considered classification (non-classification) tasks.

According to our exploratory study, we know that evaluation metrics are promising to be used as distance methods in clustering-based acquisition functions. In this part, we conduct a case study to show if it can really help improve such acquisition functions. Specifically, we modify the distance calculation method in the Coreset function from euclidean distance to BLEU and run experiments on code summarization tasks on Ruby. The reason we choose Ruby is that running evaluation metrics (especially CodeBERTScore) is time-consuming, e.g., it takes more than one month to run an active learning experiment once on code summarization tasks of JavaScript since it contains 2 times more training data than Ruby. The reason for choosing Coreset is that it performs the best in code summarization of Ruby as shown in table 5 and table 6. However, since CodeBERTScore does not support Ruby language now, we can only replace the original distance method in Coreset with the BLEU metric. Note that we still use the euclidean distance between code embeddings to compute $Pairwise\ Distances$ which is the initial step of Coreset. It is almost impossible to use the BLEU score here (the time cost is monthly). Please refer to our project site for the details of the Coreset algorithm.

Figure 3 depicts the results. We can see that Coreset with a BLEU score performs significantly better than Coreset with Euclidean distance calculated from code tokens and code embeddings. These results demonstrate the potential of using evaluation metrics as distance methods in active learning. However, using output vectors as clustering features which is also proposed by us is still the best choice which achieves the best results among all the cases. There is a big room to be improved in terms of the performance of trained models and how to propose a new acquisition function based on the evaluation metrics is still an open problem and our future research.

Recently, large deep-learning models, especially foundation models like GPT-4 (OpenAI, 2023) have gained huge attention and achieved many state-of-the-art results in various application domains. This hot trend almost changes the research focus of ML4Code from designing new code model architectures or code representation techniques to how to reuse these foundation models for our specific code tasks. Generally, model reuse involves a fine-tuning step that further optimizes the model parameters and improves performance. As a result, active code learning becomes more and more important since it allows us to fine-tune the pre-trained models with a controllable human effort, i.e., budgets allocated for labeling the datasets used in fine-tuning. This technique provides opportunities for researchers and developers with limited resources to leverage and improve existing big models.

The external threat lies in our considered acquisition functions used for active learning, code tasks and datasets, and code models. For the acquisition functions, we collect 10 functions that are specifically proposed for active learning and already studied the most in recent works (Hu et al., 2021; Weiss and Tonella, 2022). Other functions such as neural coverage methods are not considered since they are proposed for a different purpose. For code tasks and datasets, we consider both classification tasks that study important problems, problem classification, clone detection, and code summarization. For code models, we prepare two well-known code pre-trained models. Based on our open-source projects, other tasks, and models can be easily added to our benchmark. The internal threat can be the implementation of acquisition functions and code models. All implementations of acquisition functions are based on the existing active learning works (Hu et al., 2021; Li et al., 2022; Guo et al., 2022) and after carefully checking. The implementation of code models is also modified from the famous open source project (Lu et al., 2021). The construct threat can be the configuration of active learning. Since this is the first work that studies active code learning, we follow our best practice to initialize the code models and set the labeling budgets. Besides, since we compare code models under the same labeling budgets, the comparison results are not affected by the configuration of active learning.

Since active learning plays an important role in efficient model training, multiple works (Pereira-Santos et al., 2019; Settles and Craven, 2008; Yu et al., 2018; Siddhant and Lipton, 2018) conducted empirical studies on this topic. Ramirez-Loaiza et al. (Ramirez-Loaiza et al., 2017) compared different active learning methods using different measurements and concluded that improvements obtained by active learning for one performance measure often came at the expense of another measure. Heilbron et al. (Heilbron et al., 2018) studied active learning for a specific task, action localization. They found that using acquisition functions the select previously labeled data and combine them with the newly selected data is a more useful strategy of active learning for action localization. More recently, Hu et al. (Hu et al., 2021) explored the limitations of active learning. They studied adversarial robustness and the ability to handle model compression of models trained by using active learning. The results showed that models trained with active learning can achieve competitive test accuracy but suffers from robustness and compression ability loss. Michael et al. (Weiss and Tonella, 2022) conducted a replicability study and showed that the simple active learning methods, e.g., DeepGini, perform better in active learning than neuron coverage-based methods. The most recent work is (Li et al., 2022) which did a very large empirical study of 19 active learning methods including both fully-supervised active learning methods and semi-supervised active learning methods. They concluded that semi-supervised learning benefits active learning and should be considered in proposing new methods.

Different from the existing works which mainly study image data or conventional text data, our work is the first one to focus on program data which is the key type of data in the software engineering field.

ML4Code gained huge attention recently, researchers also conducted multiple empirical studies to explore the problems in the ML4Code field. In the very early work (Chirkova and Troshin, 2021), Chirkova et al. empirically study the backbone model architecture of the later code models–Transformer on different code tasks, code completion, function naming, and bug fixing. They found that Transformers can utilize syntactic information in source code to solve code tasks. Niu et al. (Niu et al., 2023) conducted a large-scale empirical study to compare pre-trained models of source code. In total, we studied 19 pre-trained models and 13 software tasks and gave fine-grained suggestions for using and evaluating these models. Steenhoek et al. (Steenhoek et al., 2022) studied the deep learning models for a specific code task, vulnerability detection. Their experimental results showed that models trained by specific types of vulnerability perform better than models trained by all vulnerabilities. Increasing the size of the training dataset has limited benefits to the performance of trained models. Jiang et al. (Jiang et al., 2023) conducted the first study of pre-trained model reuse. Concretely, they interviewed practitioners from Hugging Face and identified the challenges of pre-trained model reuse, e.g., missing attributes, discrepancies, and model risks. Besides considering only the clean performance of models of code, Mastropaolo et al. (Mastropaolo et al., 2023) studied the robustness of GitHub Copilot which is a famous code generation model. They generated semantically equivalent natural language descriptions based on the seed description and checked if Copilot can generate the same code functions as the ones generated by the seed description. They found that almost half of the semantically equivalent but different method descriptions result in different code recommendations which means the code generation models are not robust. Hu et al. (Hu et al., 2023) provided shifted datasets and studied the generalization ability of code models under data distribution shift. They found that code models are not robust and can not handle distribution shifts properly. Finally, Nie et al. (Nie et al., 2022) studied the influence of used evaluation methods on code summarization and found that different evaluation methods lead to conflicting results which should gain attention for users during testing the code models.

The above works focused on the clean performance, robustness, challenges of reuse, and evaluation methods of code models. However, non of them studied the important problem of how to reduce the labeling effort of training data and efficiently train the code models, which is our main purpose.