Code Comment Inconsistency Detection with BERT and Longformer

Natural language comments embedded within source code are indispensable to successful software projects. Although they are not compiled, comments serve as a guide to developers by providing valuable information about the functionality the source code implements as well as how developers are meant to utilize the code (Tan et al., 2012). Comments additionally help reduce ramping-up time, which is the time required for new developers to get accustomed to the specifics of a software project. Moreover, since developers on average spend around 82% of their time on program comprehension and navigation (Xia et al., 2018), maintaining consistent comments can increase developer productivity. As such, it is essential for developers to keep comments up to date with the code so that their benefits can be capitalized on. In practice, however, comments are not always updated alongside code changes (Wen et al., 2019), which can lead to them becoming outdated and inconsistent with the code. Figure 1 shows an example of an inconsistent comment versus a consistent comment. If not addressed in a timely manner, these inconsistencies can linger in code bases and mislead developers, becoming sources of confusion, bugs, and an overall impaired software development cycle (Wen et al., 2019; Jiang and Hassan, 2006; Tan et al., 2007; Ibrahim et al., 2012). Code comment inconsistency detection is therefore of immense practical use to software developers who have a vested interest in keeping their code bases easily readable, navigable, and as bug-free as possible.

Prior work focuses on rule-based techniques for specific types of comments such as references to renamed identifiers (Ratol and Robillard, 2017), API directives pertaining to parameter constraints (Zhou et al., 2017), and null values and exceptions (Tan et al., 2012). Others have attempted text similarity methods (Rabbi and Siddik, 2020; Corazza et al., 2018). However, these previous approaches only apply to pre-existing inconsistencies within a software project, which is not helpful for developers who would benefit more from a comprehensive, real-time comment maintenance system.

To resolve this dilemma, Panthaplackel et al. (2021) develop a state-of-the-art deep learning approach that not only detects existing inconsistencies (referred to as post hoc) but also flags inconsistencies that appear immediately as a result of code changes, before they can cause harm to the code base (referred to as just-in-time). A comment update model, designed to aid developers in fixing discrepancies by suggesting updates to those comments that are labeled as inconsistent, is also presented within an extrinsic evaluation (Panthaplackel et al., 2021).

In this paper, we alternatively propose a much more straightforward architecture and methodology. Our approach leverages existing large, pretrained models, namely BERT (Devlin et al., 2019) and Longformer (Beltagy et al., 2020) to address the problem of code comment inconsistency detection as a natural language inference (NLI) task.¹¹1Implementation is available at https://github.com/theo2023/coco-bert-longformer. It is well known that such models have previously demonstrated high performance on many downstream natural language processing (NLP) tasks, including NLI. To compare with the state-of-the-art, we evaluate our models on the corpus provided by Panthaplackel et al. (2021) and show that in the post hoc setting, our approach outperforms the baselines by substantial margins but underperforms the state-of-the-art in the just-in-time setting.

Our main contributions are that we (1) formulate code comment inconsistency detection as an NLI problem via binary classification; (2) show that large, pretrained language models can provide significant improvements in performance in the post hoc setting and comparable results to the state-of-the-art (without the addition of explicit features) in the just-in-time setting; and (3) offer proposals for future work in inconsistency detection and comment updating based on our approach.

Rabbi and Siddik (2020) introduce a siamese recurrent neural network architecture that detects inconsistencies by measuring the similarity between the comment and the code. They employ two separate LSTM (Hochreiter and Schmidhuber, 1997) models, one for the comment and the other for the code, and compute the Euclidean norm distance between the final hidden vectors of each LSTM. If the similarity score is below a certain predefined threshold, the comment is classified as inconsistent. While this technique achieves high recall on a limited set of Java code-comment pairs, it does not consider changes between different versions of the code and thus is incompatible with the just-in-time setting. Additionally, in general, text similarity is only a relevant metric for comments that summarize their code methods, not comments that document more specific aspects of the code (e.g., @return and @param Javadoc comments) (Panthaplackel et al., 2021).

Liu et al. (2018) develop several random forest classifiers and use 64 features relating to the code before and after modification, the comment, and their relationship. The features include code refactorings, length of the comment, and code-comment similarity, among others, in order to detect inconsistent block and line comments just-in-time. We avoid such extensive feature engineering in our procedure and also consider Javadoc comments instead of the more low-level block/line comments.

Instead of a model based on code-comment similiarity or thorough feature extraction, Panthaplackel et al. (2021) learn the relationship between comments and code modifications through a more intricate architecture that features several BiGRU (Cho et al., 2014) encoders and multi-head attention (Vaswani et al., 2017) layers. This architecture is explained in more detail in Section 5.1. Rather than using external attention modules, we rely on the built-in attention mechanisms of BERT and Longformer to correlate the comment and code tokens.

Automatic comment updating has also been of interest in the recent literature. This is the more challenging problem of resolving inconsistencies once they have been detected. Both Panthaplackel et al. (2020) and Zhu et al. (2022) develop an encoder-decoder architecture that examines changes in the source code and generates an edit action sequence specifying how the comment should be updated, which is then parsed into the gold comment. Although we focus on inconsistency detection and do not provide a comment update model in this work, we encourage further research in this important area in Section 8.

The task of code comment inconsistency detection is presented as follows: given a comment $C$ and its corresponding code method $M$, determine whether $C$ is inconsistent with $M$. We accomplish this task in two settings, post hoc and just-in-time, and formulate it in terms of NLI.

We use the training, validation, and test data curated by Panthaplackel et al. (2021).³³3https://github.com/panthap2/deep-jit-inconsistency-detection This data consists of @return, @param, and summary Javadoc comments paired with their corresponding Java methods, totaling 40,688 examples (see Table 1). They consider comment-method pairs $(C_{1},M_{1}),(C_{2},M_{2})$ from consecutive commits of popular open-source Java projects and set $C=C_{1}$, $M_{old}=M_{1}$, and $M=M_{2}$. Only examples in which $M_{1}\neq M_{2}$, i.e., where the developer modified the code method, are extracted. The data further includes a sequence of span diff subtokens for each method that we use as $M_{edit}$ in the just-in-time setting.

Collected examples are given a positive label, i.e., $C_{1}$ is inconsistent with $M_{2}$, if $C_{1}\neq C_{2}$. The assumption behind this decision is that the developer updated the comment when they modified the code because the comment would have become inconsistent otherwise. On the other hand, collected examples are given a negative label, i.e., $C_{1}$ is consistent with $M_{2}$, if $C_{1}=C_{2}$. The assumption behind this decision is that the developer did not need to update the comment when they modified the code because the comment remained consistent (Panthaplackel et al., 2021).

However, this data collection procedure introduces noise in the form of mislabeled examples (Panthaplackel et al., 2021). A positive mislabeling occurs if the developer simply improved the comment without changing its semantics; hence, $C_{1}\neq C_{2}$ even though the comment is actually consistent. A negative mislabeling occurs if the developer neglected to update the comment alongside the code changes; hence, $C_{1}=C_{2}$ even though the comment is actually inconsistent. To reduce this noise, Panthaplackel et al. (2021) only consider the 1,000 best maintained Java projects and curate a smaller clean test sample separate from the full test set. As these clean examples were not provided, however, we do not report results from that data and only evaluate on the full test set.

Lastly, comment, method, and edit action sequences are tokenized via the subword tokenization algorithms of BERT and Longformer, i.e., WordPiece (Schuster and Nakajima, 2012) and byte pair encoding (Sennrich et al., 2016) respectively.

We first discuss the baselines then introduce our models.

We report precision, recall, F1, and accuracy for our post hoc and just-in-time models in Tables 4 and 5 respectively. In the post hoc setting, we find that both models outperform the baselines with respect to precision, F1, and accuracy, and $\textsc{Longformer}(C,M)$ achieves higher scores across all metrics. In the just-in-time setting, we find that the recall achieved by $\textsc{Bert}(C,M_{edit})$ of 78.3 matches the highest (from the baseline $\textsc{Graph}(C,T_{edit})$ + features); otherwise, our just-in-time models overall underperform the Panthaplackel et al. (2021) models that have added surface features like part-of-speech tags, comment/code overlap, etc. However, these models produce results comparable to the Panthaplackel et al. (2021) models without these features.

The results in Table 4 indicate that BERT and Longformer are clearly advantageous for post hoc inconsistency detection when finetuned on an NLI task. The substantial performance gains attained by our Longformer model are foreseeable given its linear attention pattern and ability to adequately process longer sequences. However, the decrease in performance from $\textsc{Longformer}(C,M)$ to $\textsc{Longformer}(C,M_{edit})$ in the just-in-time setting (Table 5) is puzzling, especially considering the increase in performance from $\textsc{Bert}(C,M)$ to $\textsc{Bert}(C,M_{edit})$. The reason for this performance drop is unclear.

Provided these results, we leave it to future work to both improve code comment inconsistency detection as an NLI task and apply our approach to a comment update model as mentioned in Sections 1 and 2. Accordingly, in addition to the just-in-time system flagging the inconsistent comment and notifying the developer that they should manually correct it, the high-level overview is that the code changes that caused the inconsistency would be examined and suggestions would be generated to help the developer rectify the outdated comment.

Certain variants of these pretrained models also show promise. For instance, GraphCodeBERT (Guo et al., 2021) is specifically designed for processing programming language and is pretrained on a large corpus of comment-code pairs. The main contribution of GraphCodeBERT is that it takes into account the semantic structure of code through its use of data flow. Furthermore, Longformer Encoder-Decoder (Beltagy et al., 2020) is a possibility for a comment update model since it is intended for sequence-to-sequence tasks with longer input.

Another option as an implementation detail is to handle longer sequences via chunking, i.e., splitting the input sequence into smaller chunks and processing them separately. We did not pursue this method due to time constraints, but we believe it is worthwhile to explore.

We formulated code comment inconsistency detection as a natural language inference task and proposed two models based on BERT and Longformer. By evaluating on a known corpus of comment-method pairs in both a post hoc and just-in-time manner, we demonstrated that our models outperform several baselines in the post hoc setting and yield comparable results to the state-of-the-art in the just-in-time setting without additional linguistic and lexical features. We further discussed some ideas for future research in using pretrained language models for inconsistency detection and an automatic comment updating system.

We thank the 2022 Machine Learning in Cybersecurity Research Experience for Undergraduates (REU) program at Penn State University, funded by National Science Foundation Grant 1950491, for supporting this work.