On The Cross-Modal Transfer from Natural Language to Code through Adapter Modules

Transformers are state of the art neural network architecture that achieved the best results in many of the NL tasks (Vaswani et al., 2017). Transformer is stacks of encoders and decoders, each considered as a layer. It uses attention mechanism through the multi-head self attention sub-layer which is followed by a feed forward sub-layer. The multi-head self attention helps the model encode each word by attending to other words in the input sequence. Each of these sub-layers in each encoder has a residual connection, and a layer normalization is applied after each one (i.e., multi-head self attention and feed forward network). Bidirectional Encoder Representations From Transformers, BERT, is the predecessor of the N-PTLM used in our study (Devlin et al., 2018). BERT enables fine-tuning the model for downstream tasks with one additional output layer. After BERT, many PTLMs were introduced that are based on Transformer, e.g., RoBERTa (Liu et al., 2019), which is the main architecture for many C-PTLMs.

Adapters are small bottleneck layers that are inserted to a PTLM (mainly to a multilingual PTLM) and enable adapting a PTLM to a new language (Pfeiffer et al., 2020b). The adapters leverage a small number of parameters to adapt the PTLM. They are trained as language specific adapter modules (L-adapter) or task specific adapter modules (T-adapter). The former is trained via masked language modeling on unlabelled data of a target language of interest, and the latter optimizes a target task on labelled data. This training allows the PTLM to be adapted to unseen languages that are not covered in the PTLM. The framework for adapters that we use in our study is based on Multiple Adapters for Cross-lingual transfer (MAD-X) (Pfeiffer et al., 2020b), which uses an architecture as its basis that allow sharing of information between multiple tasks (Pfeiffer et al., 2021a). MAD-X enables adaptation to unseen languages in the PTLM “without learning expensive language-specific token-level embeddings”, due to being trained while keeping the parameters of the PTLM fixed (i.e., frozen). The overall architecture of the adapters is shown in Figure 2. The language and task adapters modules are inserted after the feed forward network, in each layer of the Transformer-based PTLM. The T-adapters are stacked on the L-adapters when they are needed/used for the downstream task. The language adapter $LA_{l}$ at layer $l$ of the Transformer is defined as

where $D\in{\mathbb{R}}^{h\times d}$, $h$ is the hidden size of the Transformer model, $d$ is the dimension of the adapter, and $D$ is the down-projection. $ReLU$ is the activation function and $U\in{\mathbb{R}}^{d\times h}$ is up-projection at every layer $l$. $h_{l}$ (output of the subsequent layer normalization) and $r_{l}$ (output of the feed forward layer) are the hidden state and residual at layer $l$ of the Transformer, respectively. During training of T-adapters, which are trained using labelled data, the parameters of the L-adapter of the corresponding language and the Transformer are frozen. The task adapters, $TA_{l}$ at layer $l$ of the Transformer model is similar to $LA_{l}$ and is calculated as below:

Invertible Adapters The invertible adapters are proposed in (Pfeiffer et al., 2020b) to deal with the mismatch between the vocabularies of the multilingual PTLM and the new unseen or low resource language. These are inserted on top of the input embedding layer and their inverses before the output embedding layer, as shown in the left part of Figure 2. Note that each language should have an invertible adapter in this framework. The function of invertible adapters are similar to language adapters, with the aim of capturing language specific transformations at token level. They are trained with language adapters using MLM on unlabelled data. This inversion enables efficient utilization of the “parameter budget”. This allows us to leverage the same set of parameters to adapt both input and output representations. Invertibility becomes crucial to ensure that the model does not overfit on the pre-training objective, i.e., MLM, when the model is fine-tuned on a task. For fine-tuning the model for a specific task, we remove the output embedding layer and its corresponding inversion layer following which we freeze the parameters of the L-adapters along with the PTLM’s parameters.

RoBERTa (Robustly optimized BERT approach) (Liu et al., 2019) is based on BERT and modifies its pre-training steps, which yields substantially better performance on all the classification tasks. It uses an MLM objective and includes longer sequences. RoBERTa is used in previous software engineering studies (Zhang et al., 2020a) and is the base model for the current C-PTLM models, including CodeBERT (Feng et al., 2020). RoBERTa is released in different model sizes, for which we use the 12 layers architecture known as RoBERTa-base.

CodeBERT is a BERT-style model that is good in understanding problems related to source code (Feng et al., 2020). CodeBERT is one of the models that is used as baseline on CodeXGLUE platform. CodeBERT uses the same architecture as RoBERTa-base and is trained on two training tasks of MLM and Replaced Token Detection (RTD) (Clark et al., 2020). There are two versions of trained CodeBERT model that are publicly available, one that utilizes MLM as its training objective (CodeBERT_MLM) and is trained on code corpus of CodeSearchNet dataset. The other model uses MLM + RTD objectives (CodeBERT_MLM+RTD) and is trained on bimodal data (i.e., code and documents) of CodeSearchNet. CodeBERT is trained on a combination of 6 programming languages from CodeSearchNet. For the cloze test, we use CodeBERT_MLM, as cloze test is a task that requires the model to predict the masked token. The CodeBERT_MLM+RTD cannot perform cloze test as the final layers include a discriminator model. For the same reason, CodeBERT authors only published the results of the MLM variant for cloze test in their work (Feng et al., 2020). For code clone detection, we use both variants of the model, and report the best results here.

These two models are chosen for comparison as they show the transferability of the adapters from natural language to programming language, RoBERTa being at one extreme and CodeBERT being on the other extreme. In addition, they both use the same architecture and this can provide a fair comparison between the models, especially for parameter and memory efficiency. The other C-PTLMs are not chosen here, as they use a different architecture, or are trained on a different dataset, or are not available as benchmark.

Adapter Training Datasets: Two datasets are used to train L-adapters, to evaluate whether the differences in the datasets could make a difference in the ability of the adapters ²²2Interestingly, the size of the dataset used for training does not necessarily affect their capability. For example, although the CodeSearchNet (CSN) has much more Java and Python samples to train adapters (Table 1), adapters trained using CodeNet (CN) achieve higher scores for CT-Min/Max (Table 3) and very close scores to CSN adapters for code clone detection (Table 4). . The first one is CodeNet from IBM (Puri et al., 2021), a large scale, high quality data. that is collected for studies of artificial intelligence for code. To train the L-adapters, we randomly split the data of each PL into 90-10 split for train and validation sets, respectively. The second dataset is CodeSearchNet (Husain et al., 2020), a joint effort from GitHub and Microsoft Research and consists of code and comment pairs in 6 languages. CodeSearchNet has pre-determined splits for train, validation, and test sets that we use in our study. L-adapters are trained on the training set of each dataset separately, and evaluated on the validation set of each dataset separately. The CN and CSN statistics are shown in Table 1.

Cloze Test (CT) is a probing experiment that is designed by authors of CodeBERT to evaluate their model’s capability in learning the linguistic information of code without modifying the model’s parameters (Feng et al., 2020). Here, given a code snippet, the task is posed as a multi-choice classification in which the model predicts the masked token of interest. CT task used here has two set ups on CodeXGLUE: CT-all and CT-Max/Min. In CT-all, the model should predict tokens in the source code, where the tokens come from the entire vocabulary. In CT-Max/Min, the tokens that should be predicted by the model come from {max,min} set. CT-Max/Min evaluates the model’s ability to understand code semantics (Lu et al., 2021). For testing a model on CT, no fine-tuning is required. Both CT-all and CT-Max/Min datasets are the combination of the validation and test sets of the CodeSearchNet data. We choose the portion of the dataset that is in Python and Java language as our study is on these languages. The number of instances for CT-Max/Min and CT-all in Python are 1,264 and 40,137, respectively. In Java, these numbers are 482 and 40,492 instances for CT-Max/Min and CT-all, respectively. CSN does not include C/C++. We tried building such dataset ourselves. But we could not find a dataset with similar vocabularies as CT in CodeXGLUE and their thresholds in C/C++.

Code Clone Detection (CCD) aims to identify similar fragments of code within a codebase, where identifying semantically similar code is the target task for evaluating some C-PTLMs (Wang et al., 2021). We utilize POJ-104 and Big Clone Bench (BCB) which are part of the Code-Code pipeline of CodeXGLUE (Lu et al., 2021). POJ-104 has C/C++ programs and aims to retrieve the top-k semantically similar codes and is evaluated using the MAP@R score (see Section 4.4). BCB intends to discern whether a given pair of fragments are semantically equivalent or not. It is a binary classification problem and is evaluated using the F1 score (see Section 4.4). As there is no Python dataset on CodeXGLUE for code clone detection, we consider the python-specific subset of the cross-language clone detection (XCD) dataset (Perez and Chiba, 2019). We refer to this as SCD-88 dataset, 88 pointing to the number of problems with several submitted solutions in Python. As the task of interest here is similar to the one used for POJ-104, we reformulate it as a retrieval task and evaluate using MAP@R score. Table 2 shows the respective splits for POJ-104, BCB, and SCD-88.

Reasons: CT evaluates the linguistic knowledge of the models which is important in tasks such as name prediction and code summaries. CCD is chosen as a practical Software Engineering problem. Other code-to-code tasks from CodeXGLUE require the same Language adapters but different Task adapters. As the analysis remains the same and our goal is to study the feasibility of using adapters in SE, we focus on these two tasks.

Training L-Adapters: We train the L- adapters using the invertible configuration (Pfeiffer et al., 2020b). L-adapters are trained on the code corpora of CN and CSN for each of the languages separately, leading to 5 L-adapers: Python-adapter_CN, Python-adapter_CSN, Java-adapter_CN, Java-adapter_CSN, and C/C++-adapter_CN. The L-adapters are trained on using the Adam optimizer, and a learning rate of 1E-4.

Training T-Adapters: The T-adapters are trained using the configuration as introduced in (Pfeiffer et al., 2020a). We use in-batch negative sampling to train these adapters keeping in line with the experimental setup described by the authors of CodeBERT (Feng et al., 2020). To prevent the adapters from overfitting, dropout and early stopping is used. The setup for T-adapters are the same as training baselines.

Training Baselines: To maintain consistency across our evaluations, we re-evaluated the existing benchmark performances of RoBERTa and CodeBERT for CT and clone detection in our study. We confirmed our obtained results with authors of CodeBERT, and found that they are acceptable, although our results fall within 2% error rate of what they reported. Keeping in line with the benchmark experiments of CodeXGLUE, we also utilize in-batch negative sampling. The choice of hyperparameters, learning rate schedules, and optimizers remain unchanged from CodeXGLUE’s benchmarking experiments, as although we chose different hyperparameters for fine-tuning baselines, the best results were obtained with their recommended ones and we use them in our evaluations.

All experiments are conducted on Nvidia Tesla V100 32GB GPU.

Accuracy is calculated as $\frac{TP+TN}{TP+TN+FP+FN}$. Here, the TP shows the number of records that are correctly identified to belong to the positive class, FP are records that are incorrectly classified by the model to belong to the positive class, TN are records that are correctly predicted as negative examples, and FN are the records that are incorrectly predicted as negative class.

F1-Score (F1): F1 Score is the weighted average of Precision and Recall: $F1=\frac{2\cdot(P\cdot R)}{P+R}$. Here, P stands for Precision computed as $P=\frac{TP}{TP+FP}$, and R is Recall which is calculated as $R=\frac{TP}{TP+FN}$.

Mean Average Precision at R (MAP@R) (Musgrave et al., 2020) is a metric used for informative accuracy which does not have the weakness of R-Precision metric and instead, accounts for the ranking of the correct retrievals. In R-Precision, a score of $r/R$ is assigned to each query, where for each query (e.g. a code that we want to find similar code samples), we find the r nearest samples to the query that are in the same class as query from a total number of references, R. Here, R denotes the total number of references in the searchable dataset. So, MAP@R calculates the Mean Average Precision with the number of nearest neighbors for each sample set to R. For a single query it is defined as follows where $P(i)$ is precision at $i$ if the $i$th retrieval is correct and $0$ otherwise:

For this task, neither the L-adapters, nor RoBERTa and CodeBERT are fine-tuned. Just the trained models are evaluated for accuracy on CT, which evaluates the performance of L-adapters in capturing the representation of the programming languages. The results are presented in Table 3. The rows L-adapter_CN and L-adapter_CSN represent the Python-adapter or Java-adapter that are trained on CodeNet or CodeSearchNet and are inserted in the RoBERTa model. The L-adapters are tested on the programming language that they are trained on. The models are tested in two settings, once we test them on the datasets as we obtained them. These datasets include pairs of code and a short description of its functionality in natural language. This is what originally is used by the publishers on CodeXGLUE and is shown as ‘W/ NL’ in Table 3. In the second setting, we removed the natural language descriptions and then tested the L-adapters on cloze test dataset that contains code only. This is shown as ‘W/O NL’ in Table 3. In both settings, the models should predict the masked tokens, where in the ‘W/ NL’ setting the masked tokens are from NLP and PL and in the ‘W/o NL’ setting, the masked tokens are programming language only. The is not much difference between the results of the two settings.

Note that RoBERTa is pre-trained on natural language and L-adapter_CN and L-adapter_CSN models are used to adapt this N-PTLM to the programming languages. Adapters still are able to perform closer to CodeBERT, which is fully pre-trained on programming languages (improving CodeBERT results is not our goal here). Even on CT-all task, the Java-adapter that is trained on CodeSearchNet outperforms the results of CodeBERT.

The cloze test is composed of the validation and test sets of CodeSearchNet. So, the L-adapters that are trained on CSN have better results in CT-all compared to the L-adapters trained on CodeNet. In contrast, for CT-Max/Min, the L-adaters_CN have higher scores. This can be related to the fact that CodeNet has higher number of training code samples that include the tokens min or max. This number for Python in CodeNet is 47,388, compared to 11,268 for Python language in CSN. Similarly, there are 7,640 min and max Java tokens in CodeNet compared to 5,046 in CSN. As the only prediction here is over two tokens, L-adaters_CN achieve higher scores.

The results of MODE-X for code clone detection are shown in Table 4. The programming language of the adapters in MODE-X is shown as superscript in the table, and the dataset that is used for training the L-adapters is shown as subscript. We followed the recommended settings of the baselines for fine-tuning. For CodeBERT, we provide the results for both variants of CodeBERT, pre-trained on MLM only and pre-trained on MLM + RTD.

For the T-adapters in natural language, it is reported that the last layer of the model learns the MLM objective (Pfeiffer et al., 2021b). So, they report that better results are obtained when L-adapters are dropped from the final layer, leaving only T-adapters in this layer. Therefore, we ran different experiments i) when we do not drop the L-adapters, ii) when dropping the L-adapters from layer 12, and iii) dropping L-adapters from layers 11 and 12. We report the best scores obtained. Similar to NLP adapters, the best results are obtained when L-adapters are dropped from the last or the last two layers. For BCB and SCD-88, the best scores are for the model with dropped L-adapter from its final layer, which has less than one score difference in other settings. The best results achieved for POJ-104 is by dropping the L-adapters from the last two layers, improving the results of all-layers setting by less than 3 scores.

For all three datasets, scores of MODE-X are between RoBERTa and CodeBERT that are fully fine-tuned on code clone detection, even surpassing the results of CodeBERT_MLM for BCB dataset. For C/C++ and Python datasets, the adapters’ scores are 4-5 MAP@R scores below CodeBERT_MLM+RTD. An interesting observation is that CodeBERT is not pre-trained on C/C++, but on other programming languages, and is only fine-tuned to C/C++ for clone detection task. The higher score of CodeBERT in this case is related to its learned knowledge from other programming languages. In comparison, RoBERTa have not seen any programming language during pre-training. But, adding the C/C++-adapters to its layers helps improve the model’s results for code clone detection. For Java language, adding Java-adapters to RoBERTa model improves the results of RoBERTa, which even is better than CodeBERT_MLM and is very close to CodeBERT_MLM+RTD. Note that Java is among languages that CodeBERT is pre-trained and fully fine-tuned on.

CodeBERT_MLM+RTD is trained on bimodal data, where the RTD objective is trained exclusively on source code, for all six programming languages of CodeSearchNet. So, the RTD explicitly injects source-code information into CodeBERT’s representation space. Although the impact of this dual objective may be unclear for code clone detection, CodeBERT_MLM+RTD achieves higher results over CodeBERT_MLM for other tasks (Feng et al., 2020). In our study also, CodeBERT_MLM+RTD has higher scores for two of the datasets compared to CodeBERT_MLM. In this study, we focus on evaluating the effectiveness of the cross-modal transfer abilities of the adapters. Therefore, we train the adapters solely on the MLM objective, hence, ideally would compare the adapters with CodeBERT_MLM. However, we provided the results for both variants for clarity. The MODE-X results are closer to the ones from CodeBERT_MLM, while being 60-140 times more parameter efficient in fine-tuning the parameters.

Another point to add here is that for training the T-adapters, we used the recommended hyperparamters on AdapterHub (Pfeiffer et al., 2020a). However, those hyperparameters are recommended for natural language. Therefore, we ran other experiments only for code clone detection on SCD-88 dataset. When different learning rates are used ($5E-4$ here), the results of MODE-X are improved to 79 MAP@R, which is comparable to CodeBERT_MLM+RTD and exceeds the results of CodeBERT_MLM.

The efficiency of the adapters are evaluated using i) their parameter budget, and ii) their memory usage. The parameter budget is the number of learnable parameters in the model. For adapters, as we do not re-train RoBERTa, the parameter budget is the number of parameters required for training adapters only. We report the memory and parameter budgets of the adapters for the entire 12 layers of the model, not a single adapter. Note that the numbers are the same, independent of the dataset they were trained on, as the architecture is the same. Figures 1, 4, and 5 show the parameter efficiency of the adapters compared to CodeBERT in millions.

The parameter budget for CodeBERT is 124.65 million, as it re-trains all the parameters of RoBERTa. The parameter budget for CodeBERT that is used for code clone detection is given by summing the number of parameters tuned for pre-training the model ($\sim 124$ million) and number of parameters for fine-tuning the model along with the task-specific head for code clone detection ($\sim 125$ million), adding up to 249.3 million parameters for clone detection on POJ-104 and SCD-88, and 250.48 million for BCB clone detection. This difference in the number of parameters for datasets in code clone detection is related to different formulation of this task for BCB compared to the other two.

The L-adapters and T-adapters have parameter budget of 7.39 and 0.89 million, respectively. Therefore, for code clone detection on POJ-104 and SCD-88, the number of parameters required for MODE-X is 8.28 (= 7.39 + 0.89). For code clone detection in BCB dataset, MODE-X requires more parameters, total of 9.46 million parameters. For task-specific fine-tuning, we only consider the parameters that are required for fine-tuning in CodeBERT and the parameters for training T-adapters in MODE-X (i.e., excluding the pre-training parameters of CodeBERT and parameters of L-adapters). For task-specific fine-tuning, adapters are 60.7 (= (250.48-124.65)/(9.46-7.39)) times and 140.05 (=(249.3-124.65)/0.89) times more parameter efficient than CodeBERT on BCB, and POJ-104/SCD-88, respectively. When considering the overall budget, i.e. the number of parameters required for training and fine-tuning CodeBERT and the number of parameters used for training L-adapters and T-adapters, adapters are 26.47 to 30.1 times more parameter efficient than CodeBERT for code clone detection, and 16.86 times more parameter efficient than CodeBERT for cloze test task, as CT does not require fine-tuning³³3CodeBERT is pre-trained using RoBERTa as initialization. If adding the cost of training the RoBERTa for the adapters to the parameter budget, we must do so for CodeBERT. Instead, we consider that both approaches use RoBERTa as initialization and describe the parameter budget as the total number of parameters trained..

The memory usage of the adapters and CodeBERT are shown in Table 5. The “Memory” column represents the additional memory required for a new task. The “% Model” column shows the additional memory usage over the RoBERTa model as a fraction of its memory budget. For example, when CodeBERT is fine-tuned for a new task, the whole model which is 477.98 MB should be saved. This is compared to the required memory for MODE-X, 31.63 MB, which sums up the memory usage of L-adapters and T-adapters. For CodeBERT, the whole model should be saved again (100%), which is compared to 5.28% (=(28.20/477.98)*100 ) of the RoBERTa model for L-adapters and less than one percent for T-adapters. As can be seen, MODE-X is over 15 times more memory efficient than CodeBERT, as in contradiction with the pre-trained model, CodeBERT, during fine-tuning only the adapters need to be loaded in memory.

It is worth mentioning that pre-training CodeBERT needs 384 hours of training on 16 interconnected V100s for a batch size of 64 (Feng et al., 2020). In contrast, L-adapters need 35 hours of pre-training on a single V100 GPU for the same batch size (mentioned in supplementary materials). Moreover, fine-tuning CodeBERT required 2.5 hours for CCD and less than an hour for T-adapters. As adapters are significantly more parameter efficient, they have higher inference time, which emphasizes their usage in practice.

We study the performance of the model with L-adapters, when they are added incrementally to each layer of RoBERTa. Due to the architecture of adapters, we are not able to use adapters in each layer separately, as they should receive input from the previous layers. So, we use the incremental set up for this experiment. The accuracy scores of the model with L-adapters are shown in Figure 6. The dropped layer number on the x-axis shows that we start dropping the L-adapters from that layer. For example, for each layer i, the L-adapters are inserted to all of layers 1 to i and are dropped from layers i+1 onward in the model, which is then tested for cloze test. The left column are plots of Java-adapter and the middle column are related to Python-adapter. The results are represented for L-adapters_CN in solid blue line and the L-adapters_CSN in solid yellow line. The red and green dashed lines are the accuracy of CodeBERT and RoBERTa respectively. The ND after layer 11 in the plots stands for no-drop, meaning that L-adapters are inserted in all layers. The right column of this figure shows the average results of the L-adapters and PTLMs when tested on all of the programming languages that are available for CT on CodeXGLUE: Java, Python, Ruby, Go, PHP, and JavaScript (More in Section 6).

An interesting observation here is the difference in the behavior of the L-adapters trained on CN and CSN. When the adapters are trained on CodeSearchNet, there is a small increase in the results for CT-Max/Min until layer 10. Even for CT-all. the accuracy of the L-adapters until layer 10 is close to zero. There is an increase in the scores in layer 11, and a significant jump from layer 11 to when they are inserted in all 12 layers of RoBERTa. This plateau seen for CodeSearchNet and the sudden increase in the last layer could be related to the fact that CT tasks are from validation and test sets of the same dataset. But, we could not find an explicit explanation of this behavior, noting that we are sure about training the L-adapters (not overfitting) and the generated results. In contrary, there is an increasing trend for model with L-adapters_CN for both CT-Max/Min and CT-all. The L-adapters_CN for CT-Max/Min exceed the accuracy of RoBERTa after layer 4 and 7, in Java and Python, respectively. A similar trend is seen both for Java and Python adapters after layers 8 and 7 for CT-all. As CT is considered as a probing task, it shows that the deeper adapter modules learn better semantics about the programming language than the ones used in initial layers, which is confirmed by the increasing trend in these plots.