Learning code summarization from a small and local dataset

Machine learning applications in software engineering have been very successful in practice (e.g., Microsoft’s CoPilot) and also on a wide range of more advanced applications (Lu et al., 2021). Recently, there has been a great deal of interest in foundation models (Feng et al., 2020; Wang et al., 2021; Guo et al., 2020; Kanade et al., 2020; Ahmad et al., 2021) which subjects a very highly parametrized, high-capacity neural model to a two-phase training regime. The first unsupervised “pre-training” phase is done with an enormous corpus, using a simple fill-in-the-blanks or predict-the-next-token/sentence regime. This phase can be carried out on essentially any (unlabeled) code data harvested from the web. There is no task-specific goal here; the model simply learns the statistics of the input data. The second phase, fine-tuning, trains on-task, and requires carefully curated, consistently labeled data, consisting typically of input-output pairs reflecting good, consistent, on-task performance.

The challenges of creating well-curated, de-duplicated, and yet relevant training datasets have been described by several authors (Gros et al., 2020; Allamanis, 2019). Recent paper by Ahmed et al (Ahmed and Devanbu, 2022) and Chen et al (Chen et al., 2022) explore some of the issues that arise with finding sufficient quantities of high-quality find-tuning data. Data availability may be limited because: first, some languages (e.g., Ruby) are relatively less popular than other languages, and available high-quality data may be limited. Second, the projects in a language may be skewed towards one application domain (e.g., Javascript for the web) and thus the performance of the trained model maybe somewhat uneven. Finally, and most interestingly, when curating software engineering datasets, for on-task fine-tuning, yet another strange, unique, wrinkle arises: project specificity.

It’s well known that developers in different projects do behave somewhat differently; they use different terminology, different algorithms, and even different coding practices. As far back as 2009 (Zimmermann et al., 2009) it was observed that cross-project models don’t perform as well as in-project models on defect prediction tasks. These difficulties cross-over into language models; even the earliest paper on language modeling for code (Hindle et al., 2012) noted application-specific effects. Subsequent work by Tu et al (Tu et al., 2014) and Hellendoorn (Hellendoorn and Devanbu, 2017) noted the highly local, project- and even file-specific vocabularies of source code, and proposed ways to handle them.

This phenomenon offers an entirely new opportunity: Can project-specific training data improve performance? On the plus side, since vocabulary, coding styles etc are notoriously project-specific, training and testing on the same project should give better performance. This seems like an easy, low-hanging fruit. However, there are a couple of traps. First, when working within project, one has to be careful in partitioning training and test data, so that we only use data that would be realistically available in practice. Second, within-project data may be quite substantially limited in comparison to cross-project data. For this reason, within-project training regimes would require models that learn well from fewer samples.

For this reason, we believe that it would be useful to investigate approaches that would improve sample efficiency for the fine-tuning phase of foundation model training. By “improving sample efficiency” we mean the general goal of increasing the ability of machine-learning models to learn to perform better, with fewer training samples. For example, a model $A$ that reliably performs as well as model $B$ with much fewer training samples is a more “sample efficient” model. Sample efficient model $A$ both requires less data and potentially trains much faster: thus saving human effort, time, and energy usage. Most of all, in settings where high quality training data is not as abundant, model $A$ would be more attractive.

Finally, as a sort of “stress-testing” of the same-project tuning idea, we applied this to an extremely well-tuned code summarization model, to see if same-project training provided any improvement at all. For this we used the multilingual “PolyGlot” model, by Ahmed & Devanbu (Ahmed and Devanbu, 2022). They found that cross-project, multi-lingual training, using a very large fine-tuning set in many languages provided best-in-class performance (this “PolyGlot” model was the chart-topper on the CodeXGlue Leaderboard¹¹1 See https://microsoft.github.io/CodeXGLUE/. for a while, although CodeT5 has since reported better performance). We wondered whether even this extensively well-tuned model could be further improved on a specific project by further fine-tuning on the same project. One might expect that it wouldn’t, since it is already so well trained…but actually, it worked!

In this paper we consider same-project fine-tuning, for the task of code summarization, and make the following contributions

1. (1)

   We investigate the benefits of within-project training, using a time-series scenario: we train only on “past” data, and evaluate on “future” data. In the code summarization setting, this reflects a realistic setting where a developer asks for the summary of a piece of code, and we train only data already available that specific time point in the history of the project. We find that within-project training offers some advantages.

2. (2)

   Second, we adapt the GraphCodeBERT foundation model specifically to improve its sample efficiency for code summarization; the resulting GCBhybrid model, achieves high levels of sample-efficiency and can outperform the state of the art in some project-specific settings.

3. (3)

   We also found that the “maximalist stress test”, adding project-specific fine-tuning to the already extensively fine-tuned “PolyGlot” model, actually provides further benefits, and yields the best performance, comfortably beating the state of the art CodeT5 model overall, with statistical and practical significance (pairwise Wilcoxon test, with a difference in means, over all test samples, of about 3.7 BLEU-4; this is above the 2.0 BLEU-4 threshold difference that humans are experimentally (Roy et al., 2021) known to notice).

4. (4)

   Finally, while “PolyGlot”+same-project setting is most performant, we do find that same-project training is remarkably efficient; even the largest projects use less than 2.5% of the time taken for cross-project training, while attaining comparable performance to current state-of-the-art

The paper begins in the next section with some motivating explorations of project-specific phenomena relevant to the foundation model setting. Following that we present our methodology, followed by results, discussion, and related work. The paper ends with a brief speculation on future directions.

We begin with a brief overview of Foundation models (Bommasani et al., 2021), which are currently widely used in NLP and SE. Foundation models are trained in two stages (i.e., pre-training and fine-tuning). In the pre-training stage, we train the models with billions of unsupervised tokens to teach the models the statistics of the language in a self-supervised way, using simple tasks like auto-regressively predicting the next token, filling in a blank in context, completing the next sentence, denoising an input, etc. These tasks are performed in a multi-layer deep network; the intermediate layers thus learn a representation (“embedding”) of the salient patterns of input token sequences in the code. Later we take the embeddings of the input sequence learned in the pre-training stage and further train it with supervised data in the fine-tuning stage. This pre-training+fine-tuning paradigm was first introduced in NLP by Devlin et al. (Devlin et al., 2018). They proposed an encoder-only model BERT, pre-trained with two training objectives: Mask Language Modeling (MLM) and Next Sentence Prediction (NSP). MLM is the most effective pre-training objective for encoder-only models where the model randomly masks out a certain percentage of tokens and unmasks them. Liu et al. showed that RoBERTA outperforms BERT using only MLM as a pre-training objective with some new training strategies (e.g., dynamic masking instead of static masking of a sequence) and hyperparameters tuning (Liu et al., 2019). BERT-style encoder-only models have inherent limitations for seq2seq generative tasks like Neural Machine Translations (NMT) because of the missing trained decoder. Two models, BART (Lewis et al., 2019) and T5 (Raffel et al., 2019) have well-trained decoders and perform well on seq2seq generative tasks.

These models are not designed for code; subsequent research has refined these models (retaining the same basic scheme) for code and code-related natural language description. CodeBERT (Feng et al., 2020) and GraphCodeBERT (Guo et al., 2020) are similar to the BERT model pre-trained with MLM and some code-specific pre-training objectives. PLBART (Ahmad et al., 2021) and CodeT5 (Wang et al., 2021) are replications of BART (Lewis et al., 2019) and T5 (Raffel et al., 2019) specially designed for SE tasks. Code-specific pre-trained models perform quite well on several SE tasks, including code summarization. The standard benchmark dataset CodeXGLUE (Lu et al., 2021) is used to evaluate these models. The CodeXGLUE includes a de-duplicated code summarization dataset prepared by modifying the CodeSearchNet (Husain et al., 2019) dataset. CodeXGLUE is a multilingual dataset (consisting of data from six languages) and has between 25K and 252K cross-project training samples for each language. Though some languages have relatively smaller samples (i.e., Ruby and JavaScript), other have very large training datasets (i.e., Java and Python).

Ahmed and Devanbu (Ahmed and Devanbu, 2022) have recently shown that multilingual training is beneficial for code summarization. Identifiers play a significant role in ML-based summarization, and they are mostly preserved across languages; this phenomenon enables cross-language training to work well. This prior work suggests that if methods from the same projects share similar identifiers, then same-project training can benefit the model. However, there are some issues when using same-project data. First, to be realistic, we can only use data as it becomes available; thus at any point time, only past data in the same project is available; we cannot use data on classes, methods, etc that haven’t been created yet. Thus we perform all our evaluations below in a “time-series” or “time-partioned” setting. Second, and following from this time-partitioned train-and-test approach, sample sizes get limited. Some times there is no more than a few hundred samples for each project, which differs greatly from the cross-project, cross-language setting where hundreds of thousands of instances can be used to fine-tune the models. If the pre-trained models are sample-efficient, then same-project training can be proven effective for code summarization task. We will briefly look into the two preliminary, motivating questions (PMQs) :

PMQ 1:

Do the different samples in the same project share more identifiers than samples drawn from random projects? If this were the case, one might hope that training with same project data would be especially advantageous.

PMQ 2:

Are the high capacity pre-trained models sample efficient? If these models were not especially sample-efficient, then (because same-project data might be as abundant) we might have difficulty exploiting any same-project data synergies.

1. (1)

   Divide the first 200 samples of each project into two groups (i.e., 1-100 and 101-200).

2. (2)

   Take each group and find the unique case-insensitive identifiers of the group. We repeat it for all five projects.

3. (3)

   Take group I of project A and calculate the Jaccard index with group II of the same project.

4. (4)

   Now pair group I of project A with group II of all other projects and calculate the Jaccard index²²2Jaccard Index is calculated as $\frac{\mid X\cap Y\mid}{\mid X\cup Y\mid}$.

5. (5)

   Repeat steps 3 and 4 for all projects and observe the Jaccard Index difference.

Table 2 shows that we get the highest Jaccard indices (always 2-5 times higher than other positions) in the diagonal position where the data of both groups are coming from the same projects. Hence, same-project samples have higher identifier similarities.

In the same-project fine-tuning scenario, this situation is quite common, in many projects, as we argue later. Happily, our GCBhybrid model is highly sample efficient, and attains two-digit BLEU-4 even with ten examples. This is because GCBhybrid’s pre-trained decoder is especially trained to generate (denoised) comments. GCBhybrid dominates the CodeT5 until 150 samples become available.

The following sections will discuss same-project training for code summarization using time series data and observe whether it can outperform the cross-project training performance with a few examples.

This section briefly describes the dataset preparation and foundation models we used for the evaluation.

In this section, we evaluate same-project training for the code summarization task, in different settings.

During fine-tuning, the cross-project models stop improving after 10 epochs, but same-project models continue improving even after 20 epochs, because of the smaller training set. Therefore, we fine-tune the same-project models for 30 epochs. Following all the code relevant foundation models (Ahmad et al., 2021; Feng et al., 2020; Wang et al., 2021), we use smooth BLEU-4 (Lin and Och, 2004) as the evaluation metric.

In this section, we will discuss the feasibility of applying same-project training and the computational cost needed for such training. We will also present one motivating example at the end of this section.

We can closely relate the code summarization task to Neural Machine Translation (NMT) (e.g., translating English to German). In NMT, an encoder-decoder-based framework is used to do the translation task. Reachers in the SE domain have also adopted such a framework for code summarization tasks. Systems like CodeNN (Iyer et al., 2016), DeepCom (Hu et al., 2018), Astattgru (LeClair et al., 2019), Rencos (Zhang et al., 2020), NCS (Ahmad et al., 2020) and many more applied different kinds of deep learning architecture (e.g., LSTM (Sutskever et al., 2014) and Transformers (Vaswani et al., 2017)) on encoder-decoder framework and show good performance on code summarization task. Prior work (Roy et al., 2021; Shia et al., 2022; Gros et al., 2020) discuss the evaluation metrics and datasets that have been used for code summarization task.

The main threats to our results arise from our evaluation approach. We explore some of them below

The existence, and impact, of Project-specific phenomena in software projects has been known for quite a while. The advent of foundation models, which can be fine-tuned on-task, offers a possible direction to exploit project-specificity for better on-task performance. We explore this direction, for the code summarization task, with several popular foundation models. We find that same-project training helps models exhibit competitive performance in several settings. In particular, we develop a new kind of GraphCodeBERT model, named GCBhybrid, which combines GraphCodeBERT with a specially trained decoder. GCBhybrid exhibits very high sample-efficiency, which further enables exploitation of project specificity; except “polyglot”, GCBhybrid does achieve state-of-the-art in some realistic same-project time-series settings. We also find that same-project training offers substantial savings in computational cost. In addition to code summarization, project-specific fine-tuning is a general idea that could well prove an useful adjunct for other tasks, such as defect prediction, fault localization, de-obfuscation, or automated patching. Finally, the same-project code summarization dataset and GCBhybrid source code are made available anonymously at https://doi.org/10.5281/zenodo.6523229.