Improved Automatic Summarization of Subroutines via Attention to File Context

As mentioned above, source code summarization has a long history in software engineering literature, generally following the pattern of heuristic-based methods giving way to more recent data-driven methods. Specifically, there are five very closely related data-driven papers on source code summarization which we cover in detail.

The first of the five closely-related papers is by Iyer et al. (Iyer et al., 2016) published in 2016. This work was one of the earliest to use a neural encoder-decoder architecture for code summarization. The work set the foundation for significant advancements, but in retrospect was a fairly straightforward application of off-the-shelf NMT technology: C# code was used as an input language and summary descriptions used as an output. The paper made several changes to the input during preprocessing in an attempt to maximize performance by focusing on the important parts of the code. In a sense this could be thought of as a bridge between pre-2016 heuristic-based approaches and later “big data” solutions: heuristics were used to select important words from code in preprocessing, then fed to an encoder-decoder system that, in it’s overall structure, remained unchanged from NMT approaches for natural languages.

Hu et al. (Hu et al., 2018) in 2018 proposed an improvement using the Abstract Syntax Tree (AST) of the function. Their idea was that the AST should give more details about the code behavior than the words in the code alone, and therefore should lead to improved prediction of code summaries. However the problem they encountered was that a vast majority of encoder-decoder models at the time relied on sequence input, while the AST is a tree. Their solution was to design a Structure-Based Traversal (SBT) which is essentially an algorithm for flattening the AST into a sequence and using the components of the AST to annotate words from the code.

Next, LeClair et al. (LeClair et al., 2019) at ICSE 2019 observed that the approach by Hu et al. blended two very different types of information (structure from the AST and language from identifier names etc.) in the same input sequence, while in other research areas such as image captioning (Hossain et al., 2019) it has been shown that better results are achieved when different input types are processed separately. Therefore, they designed a model architecture that processes the word sequence and SBT/AST sequences in separate recurrent nets with separate attention mechanisms, and concatenates the results into a context vector just prior to prediction.

Meanwhile, Wan et al. (Wan et al., 2018) report improvements with a hybrid AST+code attention approach. They also show how to use reinforcement learning to improve performance by several percent. While we do not dispute that the RL-based approach helps, we do not use it this paper because our goal is to show how file context adds orthogonal information to AST+code approaches. The RL-based approach is more like an improved training procedure, as opposed to adding new information to the model. Ultimately, we used LeClair et al. (LeClair et al., 2019)’s approach as a baseline because it is simpler (to reduce experimental variables) and slightly more recent.

At around the same time, Alon et al. (Alon et al., 2019) and Allamanis et al. (Allamanis et al., 2018) noted that while the AST is a useful addition to the model for prediction, it is not optimal to flatten the AST into a sequence, since that forces the model to learn tree structure information from a sequence. Yet these papers diverge substantially on the solution. Alon et al. extract a series of paths in the AST and treat each path as a different sequence, while Allamanis et al. propose using a graph-based neural network to model the source code. However, Allamanis et al. targeted generative models of the code itself. In this sense, Allamanis et al.’s work is representative of a variety of neural models for code representation (Yu et al., 2019; Zhang et al., 2019; Balog et al., 2017).

Beyond Allamanis et al.’s connection to code summarization with a recommendation of graph-based NNs, there is a diverse and growing body of work applying neural representations of code to several other problems such as commit message generation (Jiang et al., 2017), pseudocode generation (Oda et al., 2015), and code search (Gu et al., 2018). Due to space limitations we direct readers to peer-reviewed surveys by Chen et al. (Chen et al., 2018) and Allamanis et al. (Allamanis et al., 2017) as well as an online running survey (Allamanis, 2019). Song et al. (Song et al., 2019) and Nazar et al. (Nazar et al., 2016) describe code summarization in detail, including heuristic-based techniques (Sridhara et al., 2010; Sridhara et al., 2011; McBurney and McMillan, 2016).

The key supporting technology for nearly all published data-driven code summarization techniques is the neural encoder-decoder attention architecture, designed for Neural Machine Translation (NMT). The origin of the architecture is Bahdanau et al. (Bahdanau et al., 2014) in 2014. While the encoder-decoder design existed, that paper introduced attention, which drastically increased performance and launched several new threads of research. Since that time the number of papers using the attentional encoder-decoder design far exceeds what can be listed in one paper, with thousands of papers discussing overall improvements as well as adjustments for specific domains (Chu and Wang, 2018). Nevertheless, this history has been chronicled in several surveys (Young et al., 2018; Pouyanfar et al., 2018; Shrestha and Mahmood, 2019).

We include enough details in Section 4 to understand the basic encoder-decoder architecture and how our approach differs, but provide an overview here. In general, what “encoder-decoder” means is that an “encoder” is provided input data representing a source sentence or document, while the decoder is provided samples of desired output for that input. After sufficient training examples provided to the encoder and decoder, the model can often generate a reasonable output from the decoder given an input to the encoder. In the most basic setup, the encoder and decoder are both recurrent neural networks: the encoder RNN’s output state is given as the initial state of the decoder. During training, the decoder learns to predict outputs one word at a time based on the encoder RNN’s output state. Advancements in the encoder-decoder model usually focus on the encoder because the encoder is what models the input, and more accurate input modeling is likely to lead to more accurate predictions of output. For example, the encoder input RNN may be swapped for a graph-NN (Allamanis et al., 2018; Xu et al., 2018a, b; Chen et al., 2020).

The encoder-decoder design has found uses in a very wide variety of applications beyond its origin for translation. Another application area relevant to this paper is document summarization, in which a paragraph or even several pages is condensed to one or two sentences. Typical strategies include representing each sentence in the document with an RNN, and selecting words from these sentences in the decoder (Nallapati et al., 2016). From a very high level this strategy is relevant to our work, in that we model each function in a file with an RNN, though numerous important differences will become apparent in the next section.

The model is based on the encoder-decoder architecture. As in related work, for each function, we have a source code/text input as well as an AST input to the encoder, plus a summary input to the decoder. But, we introduce a new input called “file context”, which is the code/text from every other function in the same file. In this subsection, we discuss an overview visualized below:

The encoder can be thought of as modeling three types of input: code/text and the AST from a function, and the file context for that function. Likewise the decoder models one type of input: the summary. During training, a sample summary is provided to the decoder. During inference, the decoder receives the predicted summary so far, while the output prediction is the predicted next word in the summary (see the next subsection for training details).

Code/Text We model code/text using a word embedding space and a recurrent network (area 2). This is the same design that has been successful in several related papers. The code/text is merely the sequence of tokens from the source code for the function.

AST As discussed above, there are three ways related work models the AST: by flattening the tree to a sequence (Hu et al. and LeClair et al.), by using paths of the AST as separate sequences (Alon et al.), or by using a graph-NN (Allamanis et al.). We built our model so that any one of these may be used, and provide implementations for each. Later, our experiment will evaluate the effects of combining file context with each. We mention a few caveats in this section, though. First, we describe our implementation with the flattening technique shown by LeClair et al. (area 3) for simplicity and because, ultimately, we observed the highest BLEU scores in that configuration. Second, for the AST encoder by paths, we were only able to use 100 paths, which is at the lowest limit recommended by Alon et al. (they found optimal setting was 200) – when combining file context, we hit a known limit of CuDNN for large recurrent nets¹¹1https://github.com/tensorflow/tensorflow/issues/17009. Third, using graph-NNs expands the network size by tens of millions of parameters, leading to limitations on speed and memory usage; we chose two “hops” in the graph-NN as a compromise.

File Context This input leads to the key novelty of our model. The input itself is, essentially, the code/text data for every other function in the same file as the function we are trying to summarize. The format of this input is an $n$x$m$ matrix in which $n$ is the number of other functions in the file and $m$ is the number of tokens in the function (technically, we pad or truncate functions at $m$ tokens so all input sequences are the same length, and select only the first $n$ functions from each file if more then $n$ are available). We model the input first by using a trainable word embedding space (area 1). Note that we share this embedding space with the code/text data – there is not a separate vector space for words in the code/text input and words in the functions in the file context. Note that this necessitates the use of the same vocabulary for both inputs, which has the effect of introducing more out-of-vocab words for the code/text input (since vocabulary is calculated as the $x$-most common words, and the file context distorts the word occurrence count in favor of words that occur in the file context). Yet in pilot studies we found an improvement of over 20% in BLEU score when sharing this space opposed to learning different embedding spaces, not to mention reduced model size and time to train. For reference, we used a code/text vocab size of 75k, and about 15% of this was displaced when recomputing the vocabulary with file context.

Next (area 5), we use one RNN per function in the file context. This is similar to how function code/text is modeled (area 2), except that for file context we only use the final state of the RNN. For function code/text (area 2), we output the RNN state for every position in the code/text sequence – that way, we can compute attention for every position in the decoder (area 4) to every position in the code/text sequence (as described for NMT by Bahdanau et al. and implemented in a majority of code summarization papers). In contrast, for file context, we build a two dimensional matrix in which every column is a vector representing a function in the file context (the vector is the final state of the RNN for that function).

As mentioned, we calculate attention (area 6) for each encoder input to the decoder (“summary”) input. For code/text and AST sequences, we compute attention as mentioned in the previous paragraph. However, for file context, we compute attention from each position in the summary to each function in the file. A metaphor to NMT is that the attention mechanism was originally designed for building a dictionary between words in one language to words in another language, by learning to emphasize the positions in the encoder and decoder sequences that have the same meaning. Essentially what this does is train the model to output a word in one language e.g. Hund when it sees e.g. dog in the input. Applied to our model for file context, the model learns to output words in code summaries when it sees functions relevant to those words. So for instance, if a function in the file context involves playing mp3 files, the model will be more likely to output words related to mp3s, music, audio, etc. In our evaluation (see RQ₂), we explore evidence of how our model actually does behave as we envision.

To create an output prediction, we concatenate the attention-adjusted matrices from all three encoder inputs and the decoder input and use a fully-connected network (area 7) to learn how to combine the features from each input. This part of the model is similar to most encoder-decoder networks, and ultimately outputs a prediction for the next word in the summary.

We train our model using the “teacher forcing” procedure described extensively in related work (Doya, 2003, 1992). In short, this procedure involves learning to predict summaries one word at a time, while exposing the model only to the reference gold set summaries (as opposed to using the model’s own predictions during training). So for example, for every function, we train the model by providing the three encoder inputs, plus the decoder summary input one word at a time, e.g. a sample:

Ψ[ encoder inputs ] => play mp3 files
Ψ

would become during training four separate samples:

Ψ[ encoder inputs ] => <st> + ( play )
Ψ[ encoder inputs ] => <st> play + ( mp3 )
Ψ[ encoder inputs ]Ψ=> <st> play mp3 + ( files )
Ψ[ encoder inputs ]Ψ=> <st> play mp3 files + ( <et> )
Ψ

where $<st>$ and $<et>$ are start and end sentence tokens (for readability, we do not show padding). During training, the encoder would receive the encoder inputs, the decoder would receive the sequence so far (e.g. “play mp3”), and sample output prediction would be the correct next word (e.g. “files”) whether or not the model actually would have succeeded in making that prediction.

We used the corpus provided by LeClair et al. in a NAACL’19 paper of recommendations for code summarization datasets (LeClair and McMillan, 2019). This corpus includes around 2m Java methods paired with summaries, already cleaned and split into training/validation/test sets according to a variety of recommendations. That paper evaluated four different splits and determined minimal variations in reported results after cleaning. For maximum reproducibility, we use “split set 1” from that paper. We do not use datasets from other papers because we could not verify that they followed the dataset recommendations such as removing auto-generated code.

However, the corpus did not include file context (only code/text and AST) for each Java method. Therefore, we obtained from Lopes et al. (Lopes et al., 2010) the raw data use by LeClair et al., and created the $n$x$m$ file context matrices for each method (see paragraph 5 of Section 4.1). Note that each file context matrix did not include the function itself – only the other methods in the file. We included these methods even if they did not have summaries of their own. We filtered all comments and summaries from the file context so that it only included words from the code itself. In practice it may be desirable to include these comments, but we felt that including them would create a possibility that the model could see a very similar or even identical summary during training, and we decided to avoid the possibility of introducing this bias.

Note that the size of $n$ and $m$ become important hyperparamters in our approach: $n$ controls the number of functions per file, and $m$ controls the number of tokens per function. Ideally both numbers would be very high, but hardware and software limitations require us to cap them. If $n$ is too high, many functions will be included, but they will all be very short (low $m$). If $m$ is too high, we will only be able to model a few functions per file. Ultimately we tested several values and found $n$=20, $m$=25 to provide a reasonable balance. Note that the average number of methods per file in the corpus was about 8, and $n$=20 covers all functions in over 97% of files. Interestingly, performance plateaus after $m$=25. The model appears to use the function signatures and first few tokens, but later parts of the function do not appear to be useful in the file context, at least with the type of RNNs we used in our implementation.

Inspired by successful examples set by previous work, we discuss our model details in the context of our implementation source code. We built this model in a framework provided at ICSE’19 (LeClair et al., 2019), and is readable via file atfilecont.py in our fork of that framework (with a few minor edits for readability, see details Section 7).

tdat_input = Input(shape=(self.tdatlen,))
sdat_input = Input(shape=(self.n, self.m))
ast_input = Input(shape=(self.astlen,))
com_input = Input(shape=(self.comlen,))

First, above, are the input layers corresponding to the code/text, file context, and AST for the encoder, plus the summary comment for the decoder. We discussed the $n$ and $m$ hyperparameters in the previous section. For the others, we chose $tdatlen$=50 for the maximum number of tokens in the code/text sequence, $astlen$=100 for the max tokens in the flattened AST sequence, and $comlen$=13 for the max words in the output summary. The parameters $tdatlen$ and $astlen$ are as recommended by LeClair et al. in their experiments and followup discussion with the authors, while $comlen$ is limited by the available corpus.

tdel = Embedding(output_dim=100, input_dim=75000)

This line creates the code/text embedding space. The space size is one 100-dimension vector for every 75k words. We chose this size as a compromise between performance and memory usage.

tde = tdel(tdat_input)
tenc = CuDNNGRU(256, return_state=True, return_sequences=True)
tencout, tstate_h = tenc(tde)

This section corresponds to area 2 of the overview figure. Note that return_sequences is enabled, meaning that the variable tencout will contain a matrix of size 50 x 256: one 256-length vector for each of the 50 positions in the code/text sequence. Note that it is not typical for the RNN output dimensions (256 here) to exceed the word embedding vector length (100) for NMT applications, though we have repeatedly found a benefit in our pilot studies for code summarization tasks.

de = Embedding(output_dim=100, input_dim=10908)(com_input)
dec = CuDNNGRU(256, return_sequences=True)
decout = dec(de, initial_state=tstate_h)

Next is the decoder (area 4 in overview figure). The vocabulary size of 10908 is as provided in the corpus, though we note that it is less than the 44k reported by LeClair et al. in ICSE’19. The reason for the change seems to be a greatly increased training speed at minimal performance penalty, but it does mean that the results are not directly comparable to other papers – in our evaluation, we had to rerun the experiments with the new vocab size.

ae = Embedding(output_dim=10, input_dim=100)(ast_input)
ae_enc = CuDNNGRU(256, return_sequences=True)
aeout = ae_enc(ae)

This is the AST input portion of the decoder (area 3). Shown below is the SBT (Hu et al., 2018) flat AST technique, but in our experiments we swap this section for other AST encoders (see Section 4.1).

ast_attn = dot([decout, aeout], axes=[2, 2])
ast_attn = Activation(’softmax’)(ast_attn)
acontext = dot([ast_attn, aeout], axes=[2, 1])
tattn = dot([decout, tencout], axes=[2, 2])
tattn = Activation(’softmax’)(tattn)
tcontext = dot([tattn, tencout], axes=[2, 1])

The above is the attention mechanism for the code/text and AST inputs. This part is traditional attention between each position in the decoder input to each position in the code/text and AST inputs.

semb = TimeDistributed(tdel)
sde = semb(sdat_input)

These two lines begin our file context portion of the model (area 5). Basically what happens is that one 25x100 matrix is generated for every function in the file context: that is, one 100-dimension vector for every one of the 25 (hyperparameter $m$) words in each function sequence. The 100-dimension vectors are from the word embedding space shared with the code/text input (variable tdel). There is one 25x100 matrix for each of the 20 (hyperparameter $n$) functions in the file, resulting in a 20x25x100 matrix as sde.

senc = TimeDistributed(CuDNNGRU(256))
senc = senc(sde)

Next we create one RNN for each of the 20 functions. Each RNN will receive a 25x100 matrix: 25 positions of 100-dimension word vectors. Note that we also built a custom TimeDistributed layer in which we passed tstate_h as the initial state for each RNN (as it is used for the decoder), but we noticed only minuscule performance differences and removed it for simplicity. The size of senc is 20x256: one 256-length vector representing each of the 20 functions.

sattn = dot([decout, senc], axes=[2, 2])
sattn = Activation(’softmax’)(sattn)
scontext = dot([sattn, senc], axes=[2, 1])

The attention mechanism for file context looks similar to the code/text and AST inputs, but has a very different meaning. Variable sattn is the result of the dot product of decout and senc. Consider the following multiplication (table format courtesy (LeClair et al., 2019)):

In the above, vector $v1$ is the 256-dimension vector representing the first position in the decoder RNN. The vector $v3$ is the 256-dimension vector representing the first function in the file context. Value $a$ is a measure of similarity between those two vectors. Vectors that are more similar will have a higher sattn matrix.

This similarity is important because we multiply sattn with senc:

Vector $v5$ is a list of similarities of position 1 in the summary to different functions in the file context. E.g., element 3 of $v5$ is the similarity of position 1 in the summary to function 3 in the file. In contrast, vector $v7$ contains all the first elements of the 256-dimension vectors representing different functions. When each element of $v5$ is multiplied to the corresponding element in $v7$, the effect is to scale the element in $v7$ by the similarity represented in $v5$. So if position 1 of the summary is very similar to function 3 of the file while not similar to other functions (i.e. element 3 of $v5$ is high while other elements in $v5$ are low), then that position will be retained while others attenuated.

Note above that we apply a softmax activation to sattn, so each of the vectors in that matrix (such as $v5$) will sum to one. If, for example, element 3 in $v5$ is 0.90 and all others sum to 0.10, then it means that the product of the multiplication of $v5$ and $v7$ will include 90% of the value of element 3, and only 10% of the value of all other elements. That product is the value $e$: it will be dominated by the values of $v7$ (first positions of the 256-dimension vectors for the functions) that are most similar to position 1 in the decoder. Likewise, $f$ will be dominated by values of $v8$ (second positions of the function vectors) most similar to position 1 of the decoder, etc. In this way, we create a context matrix scontext which includes one 256-dimension vector for each of the 13 decoder positions – each of these vectors represents the functions most relevant to the word in that position of the summary.

context = concatenate([scontext, tcontext, acontext, decout])
squash = TimeDistributed(Dense(256, activ="relu"))(context)

We concatenate the context matrices from each attention mechanism (and the decoder) into a single context matrix along axis 1, to create a matrix of size 13x1024 (since each smaller context matrix is 13x256). We then use one fully-connected layer of size 256 to squash the 1024 matrix into a lower dimension. This is common in encoder-decoder architectures in order to prevent overfitting, though it serves an additional purpose in our model of helping the model learn how to combine the information from each of the three encoder inputs.

squash = Flatten()(squash)
out = Dense(10908, activation="softmax")(squash)

Finally, we flatten the 13x256 “squashed” context matrix into a single 3328-dimension vector so that we can connect it to a fully-connected output layer. The output layer size is the vocabulary size, and the argmax of this output layer is the index of the predicted next word in the vocabulary. Note that a large number of parameters occur between the distributed dense layer and the output layer (3328 to 10908 elements is over 36m parameters). Significant time could be gained by reducing the vocab size or the size of the “squash” layer (the 1024 could be squashed to, say, 128 instead of 256), but at unknown cost to performance. In the end, we keep these values consistent across all approaches in our experiments, to ensure an “apples to apples” comparison, even if optimizations could be made depending on user circumstances.

Our hardware included two workstations with Xeon E1530v4 CPUs, 128gb RAM, and dual Quadro P5000 GPUs. Software platform included Ubuntu 18.04.2, Tensorflow 1.14, CUDA 10.0, and CuDNN 7. The implementation above is for Keras 2.2.4 in Python 3.6.

Our research objective is to evaluate whether the our proposed mechanism for including file context in code summarization improves strong, recent baselines. In particular, we aim to establish whether any improvement can be attributed to the file context, so we aim to reduce the number and effect of other factors. We ask the following Research Questions (RQs):

RQ₁:

What is the performance of our proposed approach compared to recent baselines in terms of quantitative metrics in a standardized dataset?

RQ₂:

To what extent can differences in performance be attributed to the inclusion of file context?

The rationale for RQ₁ is straightforward: to compare our approach to existing approaches. The scope of this question is to compare our work to relevant data-driven technologies, as opposed to heuristic-based/template solutions. Generally speaking, it would not be a “fair” comparison for heuristic-based techniques because a heuristic could produce a valid summary that would not be anything like the reference solution. The way to evaluate a heuristic approach is with a human study, but the scale of the dataset (e.g. over 90k samples in the test set) is much too large. Therefore we follow precedent set in both the SE and NLP research areas, and use quantitative metrics to evaluate performance in broad terms over the whole test set.

However, there are many factors that can affect performance between one data-driven approach and another. For example, the choice of exactly which type of recurrent unit to use (e.g. LSTM vs GRU, or uni-directional vs bi-directional) or the number of units in a hidden layer. We control as many of these factors as possible by configuring the approaches in as similar a way as reasonable (see Section 5.3), but there is always a question as to whether a proposed variable is actually the dominant one. Therefore, we ask RQ₂ to study how file context contributed to predictions in the model.

To answer RQ₁, we follow the methodology established by many papers on code summarization in both SE and machine translation in NLP. We obtain a standard dataset (see Section 4.3), then use several baselines plus our approach to create predictions on the dataset, then compute quantitative metrics for the output. For our approach, we trained for ten epochs and selected the model that had the best accuracy score on the validation set. Unless otherwise stated in Section 5.3, this is also how we trained the baselines.

The quantitative metrics we use are BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004). These two metrics have various advantages and disadvantages, but one or another form the foundation of nearly all experiments on neural-based translation or summarization. Both are basically measures of similarity between a predicted sentence and a reference. BLEU creates a score based on matches of unigrams, bigrams, 3-grams, and 4-grams of words in the sentences. ROUGE encompasses a variety of metrics such as gram matches and subsequences. We report BLEU (1-4) as well as ROUGE-LCS (longest common subsequence) to provide good coverage of metrics without redundancy. In cases when the reference is only three words long (the dataset has a minimum summary length of three words), we calculate only BLEU 1-3 and do not include BLEU 4 in the total BLEU score for that sentence, because otherwise the total BLEU score would be zero even for correct three-word predictions.

We answer RQ₂ in three stages. First, we provide an overview comparison of predictions by different models to determine whether the inputs (code/text, AST, file context) give orthogonal results and estimate the proportion of predictions may be helped by file context. Second, we extract specific examples to illustrate how the approach works in practice. Whenever possible, we support our claims with quantitative evidence, to minimize the potential of bias. One key piece of evidence in the examples is from the attention layer to file context (area 6 in the overview figure, scontext in Section 4.4). The attention layer shows which of the functions in the file context are contributing the most to the prediction. By showing that a prediction is improved when when a particular function is attended, we can demonstrate that the file context is the most likely explanation for the improvement. Finally, we explore evidence that it is prevalent for file context to improve predictions in a similar way as the example, and perform an ablation study in which we train and test the model using only AST and file context.

We use five baselines. We chose these baselines because 1) they are recent, 2) they represent both code-only and AST+code neural approaches that our approach is capable of enhancing, and 3) they had reproducibility packages. Space limitations prevent us from listing all relevant details, so we provide complete implementations in our online appendix (Section 7).

attendgru This baseline represents a “no frills” attentional encoder-decoder model using unidirectional GRUs. It represents approaches such as Iyer et al. (Iyer et al., 2016), but this implementation was published by LeClair et al. (LeClair and McMillan, 2019). We configure attendgru with identical hyperparameters to our approach whenever they overlap (e.g. word embedding vector size).

ast-attendgru This is the approach LeClair et al. (LeClair et al., 2019) propose, using their recommended hyperparameters. This approach is an enhancement of SBT (Hu et al., 2018), so we only compare against this approach.

transformer Vaswani et al. (Vaswani et al., 2017) proposed an attention-only (no recurrent steps) machine translation model in 2017. It was received in the NLP community with significant fanfare so, given the success of the model for NMT, we evaluate it as a baseline.

graph2seq Allamanis et al. (Allamanis et al., 2018) proposed using a graph-NN to model source code, but applied it to a code generation task in their implementation. To reproduce the idea as a baseline for code summarization, we use a graph-NN-based text generation system proposed by Xu et al. (Xu et al., 2018a). We use the nodes and edges of the AST as input, with all other configuration parameters as recommended for NMT tasks by Xu et al..

code2seq Alon et al. (Alon et al., 2019), described in Section 3.1, is a recent code summarization approach based on AST paths that is reported to have good results on a C# dataset. We reimplemented the approach from Section 3.2 of their paper, with their online implementation as a guide. Note we did not use their implementation verbatim. They had many other architecture variations, preprocessing, etc., that we had to remove to control experimental variables. Otherwise, it would not have been possible to know whether performance differences were due to file context or these other factors.

Our key finding in answering RQ₁ is that, in terms of quantitative measures over the whole test set, adding our file context encoder increased performance in nearly all cases. Figure 1 showcases the difference when compared with aggregate BLEU score (column BLEU-A of Table 1). The baselines attendgru and ast-attendgru improved by more than one full BLEU point, while graph2seq and code2seq improved by around 0.3 BLEU. One possible explanation for the greater increase for attendgru and ast-attendgru is that the path- and graph-based AST encoder models have many millions more parameters than the flattened AST approach (not to mention, when there is no AST encoder), and the model may have difficulty retaining some of the details in these larger encoders in the “squash” layer of the model (area 7 of overview figure in Section 4.1 and paragraph 13 of Section 4.4). Recall that an overriding objective in our experimental setup is to reduce variables to create an “apples to apples” comparison. For that reason, we used fully-connected networks of 256 dimensions for all approaches in the squash layer. The result could be that the model is able to learn more details about the flattened AST encoder in the squash layer simply because of there are many fewer parameters in that encoder, while we are in effect asking the model to remember much more information in that layer in the path- and graph-based encoders. Our recommendation for future work is to designate the size of the squash layer as a hyperparameter for tuning, perhaps with larger settings for larger encoders. We do not recommend concluding from this experiment alone that a flat AST encoder is the best design. Instead, we confine ourselves to the conclusion that adding our file context encoder to the baselines improves the performance of those baselines.

An exception to the overall observation of improved performance with the file context encoder is that the graph2seq model obtain a slightly lower ROUGE-LCS F1 score, despite a higher aggregate BLEU score, when we added the file context encoder. The way to interpret ROUGE-LCS is that precision and recall are calculated only for words in a predicted sentence that appear in sequence when compared to a reference sentence. For example, consider prediction “converts the file from mp3 to wav” and reference “converts the file from wav to mp3”. The summaries have very different meanings, but the BLEU1-3 scores will be fairly high and will inflate the aggregate BLEU score. In contrast, ROUGE-LCS will result in precision and recall scores of 57% (4/7 for longest common subsequence divided by sentence lengths, see Section 3 of Lin et al. (Lin, 2004) for formulae) – the result is a score much more based on proper word order instead of predicting correct words anywhere in the sentence. So, it appears that the predictions from graph2seq have slightly longer common subsequences with the references, without the file context encoder.

We made several related but tangential observations. We found that the transformer model performed quite poorly on this software dataset, despite reports of excellent performance at low cost on natural language translation (Vaswani et al., 2017; Young et al., 2018). While it is tempting to draw sweeping conclusions from this finding, we mention it only to recommend caution in applying NMT solutions to this problem. Evidence is accumulating in related literature that code summarization is not merely an application of off-the-shelf technology borrowed from the NLP domain (Hellendoorn and Devanbu, 2017). Another tangential observation is that we verify conclusions by related work, namely Alon et al. (Alon et al., 2019) that a path-based AST encoder outperforms most alternatives (code2seq was the highest performer without file context), and Hu et al. (Hu et al., 2018) and LeClair et al. (LeClair et al., 2019) that a flat AST encoder design outperforms traditional seq2seq encoder-decoder designs (i.e. attendgru). Finally, we note that the AST-based models have broadly similar performance (18.61 - 18.84 BLEU), while the contribution of file context varies much more (18.88 - 19.95 BLEU).

We explore the effects of file context in three ways: First we offer a bird’s eye view of results. Second we present specific examples demonstrating how the model works. Third we provide evidence that the examples are representative of the model’s behavior.

Overview Consider Figure 2(a) which shows a breakdown of BLEU1 scores for predictions by attendgru. (We show BLEU 1 scores for simplicity of comparison, BLEU 1 is equivalent to unigram precision.) There is a significant portion in which attendgru performs quite well, with BLEU1 scores sometimes even nearing 100. This is not surprising since in some cases, the source code of the function “gives away” the summary e.g. in functions with names like convertMp3ToWav. Yet for 46% of predictions, BLEU1 score is less than 25, meaning that not even 25% of the words in the predictions are correct. This is also not surprising, since in many cases the code has almost no clues as to what words should be in the summary.

Next consider Figure 2(b). This chart shows the percent of functions in which the predictions had the highest BLEU1 score (of predictions ¿25 BLEU1) for three different models: attendgru relies only on code/text, ast-attendgru is the same model but with access to AST information, and attendgru+FC is the same model but with access to file context (but no AST). What we observe is that there is a subset of functions for which each model seems to perform best – it is not as through the models provide uniformly better results on all functions. Related work e.g. LeClair et al. (LeClair et al., 2019) showed how AST-based models can improve predictions for functions in which the structure contains clues about the function’s behavior, even if the source code contains no useful words. We make a similar observation for file context. The attendgru+FC model performs best for a subset of around 22% of functions (and ties with ast-attendgru for 4% of functions). When combined, a major portion of the improvement comes from reducing the number of low quality predictions (visible in a reduced number of ¡25 BLEU1 scores) in addition to improving many of the predictions of the baselines. The result is the overall improvement in BLEU scores reported for RQ₁ for models that combine many types of input data such as ast-attendgru+FC.

Examples Below are two examples showing how using file context improves predictions. We chose these examples based on explanatory power: they are short methods in which ast-attendgru and ast-attendgru+FC differed by one word (there are 109 such examples out of 6945 where ast-attendgru+FC outperformed ast-atte- ndgru over the 90908 methods in the test set).

In Example 1, it is obvious from the code that the method is just a setter, but there is no hope to understand what the value means even with the AST, and the baselines output an unknown token. But, the file context reveals several keywords e.g. flight, airline, departure that serve as clues. The attention matrix shows that the model found these clues (the image shows a heatmap of the values in sattn in Section 4.4 just prior to predicting the last word). High activation is visible connecting the later positions in the output prediction to function 3, which contains the word “flight.” The effect of high activation on this function is that the context vector (which is a concatenation of the attention matrices) will be much closer in vector space to the words related to flights, airlines, etc. in the word embedding, than it would be with only the code/text and AST. This makes it much easier for the model to predict the correct word.

Example 2 is similar, except that the word that is different in the predictions (“exception” vs “object”) is not directly in the file context. Once again the source code gives few clues except that it returns a string representation of something. The baselines give the term “object” which is a reasonable guess but not specific. The attention matrix shows high activation on the three functions which contain the term “stack trace”, which is nearby to “exception” in the word embedding (technically “stack trace” is two words, but since the function representation is an RNN, the final state will contain information from both words).

Prevalence The examples above show how file context can improve predictions in specific cases, but we chose these examples based on explanatory power and not necessarily prevalence. In fact, most of the predictions are more complicated (most predictions differed by more than one word, and most methods are larger). Consider that there were 6945 methods out of 90908 in the test set where ast-attendgru+FC outperformed ast-attendgru in terms of aggregate BLEU score. Of these, 5093 (73%) had words in the reference summary that were in the file context but not the source code of the method; ast-attendgru+FC correctly used these words in the output predictions for 4369 (86%). Put another way, there were 4369/90908 (about 5%) methods in the test set where ast-attendgru failed to find the correct word, but ast-attendgru+FC did find the correct word, and that word was in the file context. Improvement over this 5% largely explains the increase in aggregate BLEU score from 18.69 for ast-attendgru to 19.95 for ast-attendgru+FC (+6.7%). However, in practice, the file context sometimes led the model astray. Of the 90908 test set methods, ast-attendgru and ast-attendgru+FC tied the aggregate BLEU score 78682 times. For the 12226 times they differed, as mentioned, 6945 times ast-attendgru+FC outperformed ast-attendgru while 5281 times it underperformed. Of these, 2761 (52%) occurred when a ast-attendgru+FC picked a word from the file context when the reference did not contain that word.

As a final piece of evidence studying the effect of file context, we perform an ablation study in which we train and test two models in an extreme condition: when zero code/text data are available. Basically we train and test with AST and file context only, code/text sequences are set to zeros. Ablation studies are common in NMT research to determine whether a given input is benefiting a model (Kuncoro et al., 2017). Our study is akin to the challenge experiment proposed by LeClair et al. (LeClair et al., 2019), but differs in that our intent is to explore file context rather than simulate code obfuscation. For brevity, we present only BLEU score values for ast-attendgru and ast-attendgru+FC:

The ast-attendgru model had only the AST from which to base predictions. As shown in the prediction for Example 1 below, it identified that the method sets some value (because it receives a parameter and sets a class property to that parameter’s value), but without any text it only predicts an unknown token. On the other hand, ast-attendgru+FC guesses terms from the file context such as “departure airport.” While not technically a correct prediction, this example illustrates how the file context can serve as an alternative to text information in the source code of the method. Overall, the aggregate BLEU score increases from 8.51 to 11.31 (33%). This difference is almost certainly due to the file context, since the code/text data are ablated and all other details of the model are identical.