Zero-Shot Code Representation Learning via Prompt Tuning

Pre-trained language models (PLMs) such as BERT (DevlinCLT19), GPT, and T5 (RaffelSRLNMZLL20) have been shown to provide large improvements for a range of natural language tasks. The key idea of PLMs is to train a large model on vast corpora and use the resulting representations on tasks for which only limited amounts of labeled data is available.

A PLM is pre-trained on a large-scale text corpora through a series of self-supervised learning tasks, e.g., masked language modeling (MLM) and next sentence prediction (NSP). The MLM task masks a random portion of tokens in the input text and tries to predict the masked words, while the NSP task predicts whether or not two given input segments are coherent. These pre-training tasks enable PLMs to learn the general knowledge from large language corpora. Then the PLM is fine-tuned on a task-specific dataset. A fine-tuning header on top of the PLM is optimized via supervised learning tasks in a specific domain. This header is usually a trainable neural network such as an MLP for classification tasks or a Transformer decoder for generative tasks.

Given the great success of PLMs in NLP, researchers also seek the adaptations of PLMs for programming languages (codebert; codet5; codex). They customize the pre-training objectives using programming related tasks on a big code corpus. PLMs have been successfully used to learn code representations and further be adapted in a variety of code intelligence tasks, e.g., clone detection (FangLS0S20), code summarization (ChoiBNL21), and code search (HaldarWXH20).

For example, CodeBERT is built on RoBERTa (roberta) and is pre-trained with both natural and programming languages. Figure 1 illustrates the main pipeline of CodeBERT. The model is first pre-trained on two tasks, namely, MLM and replaced token detection (RTD). The MLM task randomly masks the tokens in natural language and programming language (NL-PL) pairs and trains the model to predict the original words. The RTD task trains the model to detect whether a given token is original or generated by the model. Having pre-trained, the pre-trained model is fine-tuned on data of downstream tasks such as clone detection and code search. A fine-tuning header is added to the PLM and is optimized with downstream tasks.

The standard supervised learning approaches train a model with large-scale labelled samples. However, in many tasks such as recognizing name of a new brand or translating a new language, obtaining sufficient training samples is laborious and often impracticable. Zero-shot learning transfers a learned model from a source domain to a target domain that has no labelled data, and hence alleviates this “data hungry” problem. It can be realized through a variety of techniques such as data augmentation (BorneaPRFS21), meta-learning (metalearning1; metalearning2), PLMs (DevlinCLT19), and prompt-based learning (gpt3).

Data augmentation. A direct technique of zero-shot learning is data augmentation, namely, enlarging the data set (e.g., randomly inserting samples and noise) so that the model can have sufficient data samples for training (BorneaPRFS21).

Meta-learning. Another popular strategy for zero-shot learning is meta-learning. Meta learning is also known as “learning to learn”, which aims at training a meta learner which learns the update rules of the target model (GuWCLC18). This enables a machine learning model to achieve competitive performance even with scarce data. However, meta-learning focuses on learning strategies instead of representations. Hence it will be difficult to be generalized across different code intelligence tasks.

Pre-trained Language Models. PLMs are pre-trained on large-scale text corpora to learn common knowledge of the languages, and can be generalized to specific tasks with only a few training examples. However, PLMs need a fine-tuning phase which continually train the pre-trained model on the downstream tasks. Fine-tuning requires the availability of manually labelled datasets which is laborious and expensive.

Prompt-based Learning. To alleviate the data hungry problem of fine-tuning, GPT3 (gpt3) introduces prompt-based learning, a lightweight alternative of fine-tuning for PLMs. A prompt is usually a piece of text inserted in the input to guide the pre-trained model to generate desirable results. For example, one can prepend “TL;DR” followed by a few examples in the input of GPT3 to let it summarize the input.

Unlike fine-tuning, which adds fine-tuning header and re-optimizes the PLM using downstream tasks, the prompt-based learning approach converts downstream tasks (e.g., method name detection) to the same form as the pre-training tasks (e.g., MLM) by injecting “prompts” and “[MASK]” to the PLM input. Hence, the PLM can generate the desired results with minimal adjustment. This encourages downstream tasks to reuse the knowledge from the PLM more efficiently.

The learned program vectors can further be taken as input to machine learning models for code intelligence tasks, such as code clone detection, code search, and method name prediction.

State-of-the-art code representation learning techniques usually leverage the pre-training and fine-tuning paradigm: a Transformer is firstly pre-trained on large unlabeled code corpora using self-supervised objectives and is subsequently fine-tuned on labelled data of code classification tasks.

$f_{\theta}^{S}$ couldn’t be used for the representation of $\mathcal{X}^{T}$ directly since there is a lexical gap between $\mathcal{X}^{S}$ and $\mathcal{X}^{T}$. However, both the source and the target language PLMs are pre-trained on large-scale unlabelled code corpora with self-supervised objectives such as the MLM for a PLM. Hence, it is feasible to bridge their representation gap by using the common knowledge learned in pre-training. Based on this idea, we cast the problem in Equation 2 as a pre-training task for PLM and train the model on ($\mathcal{X}^{S}$, $\mathcal{Y}^{S}$), so that the PLM can also predict reasonable results for samples in $\mathcal{X}^{T}$ seamlessly.

Figure 2 illustrate the overall architecture of Zecoler. The pipeline is comprised of three steps:

• 1)

  We first cast any downstream task to the pre-training task (e.g., MLM) by inserting trainable prompts and a “[MASK]” token into the input of the task. The added prompts play the role of guiding the PLM to elicit the knowledge learned in pre-training and predict the right answer (§3.3).

• 2)

  Taking the resulting data as input, a PLM is then continually trained on the source language dataset of downstream tasks, that is, infers the code representation of the input and predicts the word for the “[MASK]” token. The optimal prompts are searched in the word embedding space automatically (§3.4).

• 3)

  Finally, we take unlabelled dataset in the target language as the input to the PLM and let the model predict the answer without training. A verbalizer is employed to cast the predicted word to class labels (§3.5).

Our first step is to cast the downstream task (Equation 2) into the MLM task. We concatenate two input snippets $x_{1}$ and $x_{2}$ of the downstream task, which can better capture the relationship between them (nspbert). Like that in the MLM task, we also insert an “[MASK]” token into the concatenated input. The “[MASK]” token acts as a placeholder which steers the pre-trained model to generate the classification result $y$ in the code intelligence task. It is notable that the position of the “[MASK]” token is a hyperparameter and we append it at the tail of the input by default. The masked sequence

is taken as input to the PLM, which yields the hidden states

for all tokens. Then, the hidden state corresponding to the masked token, namely, $\mathbf{h}_{-1}$, is fed into an MLM header $g_{\phi}$ which predicts a token for the masked position:

The MLM header $g_{\phi}$ is a fully connected neural network parameterized by $\phi$ that is optimized to minimize the cross-entropy loss:

where $\mathbf{y}$ denotes the ground-truth label of the code intelligence task, and $|V|$ is the vocabulary size.

The conventional fine-tuning method optimizes $f_{\theta}$ and $g_{\phi}$ in Equation 2 from scratch. This causes the model to overfit scarce task-specific data. Inspired by prompt-based learning (ptuning), we optimize the PLM by merely adjusting its input sequence. More specifically, we insert a number of pseudo tokens called prompts into the input sequence of the PLM, which coax the PLM to directly generate the predicted class label of the downstream task. By only adjusting the model input, the PLM needs far less optimization cost to fit for the data in the target task, while keeping the most of prior knowledge learned during pre-training.

Based on this idea, we design a number of prompt tokens $P=[P_{1},...,P_{m}]$ and inject them into the masked sequence $\tilde{x}$ using a pre-defined template $T={[P];x_{1};[P];x_{2};[P];[MASK]}$. Hence, the original inputs $x_{1}$ and $x_{2}$ are transformed into

through template $T$ which contains m prompt tokens.

Like general words, these prompt tokens are embedded into trainable vectors and are continually pre-trained on downstream tasks of the source domain through gradient descend.

However, the number of trainable parameters for prompts are too small compared with that of the original PLM. This may cause the prompt representation vectors fall into local minima in gradient descend. To solve this problem, we train prompts using a bidirectional LSTM additionally.

In a zero-shot setting, there is no training sample in the target low-resource language. Instead, we continually pre-train the PLM and the trainable prompts using large-scale code corpora in popular languages (e.g., Java), and then directly apply the trained model to tasks in the low-resource language. More specifically, we train the PLM in the source domain through prompt-based learning. Then, we take as input data samples in the target domain into the same model without extra training, and obtain the results of the downstream tasks.

The MLM task generates a token that is likely to fill into the masked position. In order to obtain the classification result, we need to revert the MLM predictions to classification labels of the downstream task. For this purpose, we employ a verbalizer (pet) which realizes such a reversion. Let $\mathcal{V}$ be the vocabulary of the PLM and $\mathcal{Y}$ be the labels of the downstream task such as $\{$true, false$\}$. The verbalizer is defined as a function $v$: $\mathcal{C}\rightarrow\mathcal{Y}$ that maps each candidate word in the vocabulary to a classification label. The choice of candidate words is arbitrary as long as they are sufficiently different. The model will be trained to map candidate words to true predictions. In our approach, we consider two candidate words $\{yes,no\}\in\mathcal{V}$ as a candidate set $\mathcal{C}$ and only inspect which word in $\mathcal{C}$ is more likely to fill into the “[MASK]” position through PLM predictions. If the word “yes” has a higher probability to fill in the masked position, the verbalizer will map it to the label “true” and hence output a positive prediction for this task.

Take code clone detection as an example. Given two code snippets, the model constructs an input sequence by injecting a number of prompt tokens into the snippets, followed by a “[MASK]” token. The constructed sequence is fed into the PLM to predict the label $Y$, where $Y\in\{$“cloned”, “not cloned”$\}$. The MLM header of the PLM outputs the probability of each candidate word for the masked position. If the candidate word “yes” has a higher probability, the verbalizer will map it to the class label “cloned”, yielding the final prediction as “cloned”.

Besides classification tasks, we also explore zero-shot learning on generative tasks such as program generation and code summarization.

where $p$ denotes the probability of $y$, $f_{\theta}$ denotes the representation learner and $g_{\phi}$ denotes a neural generative decoder. For instance, the code summarization task generates a brief summary $y$ for the given code $x$.

Figure 3 illustrates the architecture of Zecoler for generative tasks. The model follows a conventional encoder-decoder architecture. The pipeline is comprised of two steps: First, the PLM encoder takes as input a code snippet or a description $x$, prepends it with a prompt $P$, and encodes $\tilde{x}$ as vectors. Then, the decoder generates the target sentence $y$ based on the encoded hidden vectors.

The prompt $P$ is designed as a sequence of $m$ prefix tokens, which helps the PLM extract the knowledge learned in the pre-training phase. In code generation task, in order to specify the target language for guiding model to generate the correct program, we append a special token “$<$language$>$” to the original prompt where “language” indicates the target language name.

We select CodeBERT as the encoder, and a randomly-initialed Transformer as the decoder. Both the PLM and prompt are trained on the task-specific dataset in the source language and are then transferred to the generative task in the target language without extra training.

Figure 4 shows the workflow of Zecoler. Zecoler follows the general paradigm of learning code representations. In the training phase, Zecoler is given a training set of labelled code snippets. For each snippet (pair), Zecoler augments it using a prompt template. The prompt-augmented code (pair) is taken as input to Zecoler which yields the prediction and calculates the loss function based on the ground-truth data.

In the usage phase, Zecoler is given a code snippet (pair) only. Zecoler augments it using the same prompt template as in the training phase and then gives the prediction for the downstream task.

We evaluate Zecoler by answering the following research questions:

• •

  RQ1: How effective is our approach in zero-shot code representation learning?

  We evaluate the effectiveness of Zecoler in zero-shot code representation learning. We take Java as the source language and transfer the learned model to Solidity, a domain-specific languages, and Go, an up-and-coming language which are not provided with training samples. The experiments are conducted in three popular classification tasks.

• •

  RQ2: How effective is our approach in few-shot code representation learning?

  In some programming languages and tasks, we can obtain scarce (e.g., 100) samples in the target language. We wonder whether Zecoler is also effective in these data when pre-trained on the source domain. Therefore, we provide the pre-trained model in RQ1 with a few samples of the target language and conduct the same experiments as in RQ1.

• •

  RQ3: How effective is our approach in monolingual code representation learning?

  RQ1 and RQ2 mainly evaluate the effectiveness of Zecoler in a cross-language setting. We further explore how effective is our approach without transfer learning. We train the model in three languages in the few shot setting, and test the model in the same language. Besides Solidity, we also want to assess our approach in other languages such as Java and Go when training data is insufficient.

• •

  RQ4: How effective is our approach in zero-shot generative tasks?

  Besides the classification tasks, we also investigate the effectiveness of Zecoler in generative tasks. Similar to RQ1, we take Java as the source language and take Solidity and Go as the target language. We also take JavaScript and Ruby into account as the examples of program languages in the specific projects. The experiments are conducted in code summarization and code generation tasks.

• •

  RQ5: How do different hyperparameters impact the performance of our approach?

  We evaluate the performance of our approach under different hyperparameters. Specifically, we conduct ablation studies on prompt templates (number and position), source languages, and PLM scales.

We evaluate our approach on three classification tasks and two generative tasks:

• 1)

  Code Clone Detection (CD): a task that determines whether two code snippets are cloned or not (bigclonebench). A PLM based clone detection model takes as input two code snippets and outputs their representations. Then, a classification header is built on top of the representations and predicts whether the two code snippets are cloned (=1) or not (=0). There are four types of clones (clonedetection). Our approach can challenge type-3 and type-4 clones, that is, the two snippets are not textually identical, but implement the same functionality.

• 2)

  Code Search (CS): a task that retrieves a semantically relevant code snippet for a given natural language query (codesearchnet). Following CodeBERT (codebert), we formulate code search as a classification problem. Given a natural language description of the code and a programming language code snippet, this task aims at determining whether this NL-PL pair is related. The binary answer is “related” or “not related”. The classification generates a probability score, which can be used for ranking results of code search.

• 3)

  Method Name Prediction (MNP): a task that suggests the function name for a code snippet (ZhangCLP21). Similar to code search, we transform this task to a binary classification task (ComptonFPK20): given a code snippet, it enumerates all candidate function names (i.e., the vocabulary of code tokens) and constructs a “$<$snippet, name$>$” pair. The pair is taken as input to the PLM which outputs a binary prediction whether the name in the pair is “suitable” (=1) or “not suitable”(=0) for the code snippet.

• 4)

  Code Summarization (CM): a task that generates a natural language summary for given source code (00030W0Z22). We fine-tune the PLM using a parallel dataset of PL-NL pairs.

• 5)

  Code Generation (CG): a task that automatically generates source code for a natural language query (WangWWMLZLWJL22). Code generation is a challenging task because programs usually follow syntax rules while PLMs are good at generating sequential tokens. We fine-tune the PLM on a parallel set of NL-PL pairs.

We conduct our experiments on four datasets. Each dataset may be used for multiple downstream tasks. Table 1 shows the statistics of each dataset, including sizes, programming languages, and corresponding tasks.

We preprocess these datasets by removing comments from the code, since they can interfere the final results in classification task experiments. Code snippets with more than 250 tokens are filtered out to fit for the PLMs. We also exclude code snippets with less than 125 tokens to accommodate downstream tasks. In order to prevent the model from being biased to one class in classification tasks, we balance the dataset with the same number (1:1) of positive and negative samples. The negative pairs ($y=0$) are created using random combinations of snippets from the positive data samples ($y=1$).

We implement our models on top of CodeBERT, a popular PLM which is built based on RoBERTa-base (H=768, A=12, L=12). CodeBERT learns representations of programming languages (Java, Python, JavaScript, PHP, Ruby, and Go) in the pre-training phase. We use the default tokenizer (i.e., Microsoft/codebert-base) of CodeBERT with a vocabulary size of 50,265. We set the maximum sequence length to 512. Our experimental implementation is based on the Huggingface Transformers⁴⁴4https://huggingface.co/microsoft/codebert-base and P-Tuning (ptuning). The batch size and the number of epochs are set to 10 and 20 in classification tasks, and 20 and 15 in generative tasks.

In classification tasks, we insert prompt tokens uniformly into the original input of CodeBERT since there are two text snippets in the input. The additional LSTM for training prompts has two hidden layers followed by two-layer multilayer perceptrons (MLP) activated by ReLU.

To generate target sequences, we employ a 6-layer Transformer decoder as a fine-tuning header. Since we append a special prompt “$<$language$>$” to the original input and change the input pattern, we continually pre-train CodeBERT using MLM task on 100,000 unlabelled code sampled from CodeSearchNet, each appended with a language mark. The batch size and number of epochs are set to 8 and 3, respectively.

All models are optimized using the AdamW (adamw) algorithm on a machine with a GeForce RTX 3090 Ti GPUs. The initial learning rate (lr) is set to 3e-5, which linearly increases from 0 during a warm-up period. The iteration number of the warm-up period equals to the number of the training steps in the first epoch. During the rest training process, the learning rate continuously decreases to 0. We measure the performance on the validation set during training. The checkpoint that achieves the best accuracy on the validation set are selected for testing.

We compare our approach with five baseline models:

• 1)

  AVG: a baseline approach that directly represents programs by averaging their token embeddings. We reuse token embeddings from CodeBERT and represent an input code snippet by the average of all token embeddings. Next, we fine-tune the classifier of downstream tasks using a 3-layer MLP header.

• 2)

  RoBERTa (roberta): a popular pre-trained language model that has also been used for programming languages (codebert). The model is constructed with 12 transformer layers and pre-trained on a large English corpus with the MLM objective. We fine-tune it with a 3-layer MLP header over the “[CLS]” position.

• 3)

  RoBERTa-large⁵⁵5https://huggingface.co/roberta-large: a large version of RoBERTa (H=1024, A=16, L=24) with around 300 million parameters. We compare with this model to verify the advantages of Zecoler over large-scale PLMs.

• 4)

  CodeBERTa⁶⁶6https://huggingface.co/huggingface/CodeBERTa-small-v1: a version of RoBERTa pre-trained with CodeSearchNet, which was proposed by Huggingface. We use its default setting in our experiments.

• 5)

  CodeBERT (codebert): one of the state-of-the-art models for learning code representations. A more detailed description of CodeBERT can be found in Section 2.1. We follow the same experimental setup in its original paper.

We implement these baseline models by referring to the work of CodeXGlue (codexglue). In classification tasks, we construct 3-layer fully connected neural networks as the fine-tuning header which maps the hidden vector of the “[CLS]” token to the class labels of downstream tasks. In generative tasks, we use a 6-layer transformer decoder as the fine-tuning header for generating the target sequence of downstream tasks.

In this experiment, we evaluate the effectiveness of Zecoler in zero-shot code representations learning. We initially train a representation model for each task using data samples of Java. Then, we adapt the trained model to the target languages (i.e., Solidity and Go) directly without extra training. We train the model with both 5,000 and 500 data samples of Java to assess the effects under different data sizes.

Table 2 shows the accuracy of different models in three classification tasks. We can observe that Zecoler significantly outperforms baseline models in all three tasks and all target languages. In the code clone detection task, the accuracy of Zecoler is 5%-14% greater than that of CodeBERT, the strongest baseline. The improvement is much more significant in the code search (30% in average) and method name prediction (24% in average) tasks. By contrast, AVG and RoBERTa-large obtain results that are close to random, indicating that they can hardly learn useful knowledge from few data samples.

The same trend can be observed when only 500 (1/10) samples of the source language are provided for training. As the data size decreases from 5000 to 500, the accuracy of Zecoler drops in all tasks. Nevertheless, it still significantly outperforms the baseline models. This means that Zecoler can learn representations much more efficiently while requiring smaller data compared with baselines.

Another interesting observation is that Zecoler trained with 500 data samples outperforms that with 5,000 data samples in the method name prediction task for Solidity. One potential reason is that the model overfits the big Java data in the training process, which results in inferior performance in the target low-resource languages such as Solidity.

It is notable that CodeBERT outperforms RoBERTa and CodeBERTa in both code clone detection and method name prediction, except for the code search task. We conjecture that CodeBERT is pre-trained on programming languages whereas the RoBERTa is only pre-trained on natural languages. Hence, CodeBERT can be better adapted to PL related tasks.

In this experiment, we evaluate the effectiveness of Zecoler in few-shot learning of code representations. This simulates the scenario where there are a few training data samples in the target language. We continue training the model in RQ1 using a few data samples of the target languages. We vary the data sizes from 32 to 700 in Solidity and Go and evaluate the performance in three classification tasks.

Figure 5 shows the results in classification tasks. Compared to CodeBERT, the strongest baseline in RQ1, Zecoler demonstrates greater strengths in all tasks when provided with a few data samples of the target language. When the data size is 700, the accuracy of Zecoler is about 30% greater than that of CodeBERT. This means that Zecoler outperforms existing approaches in learn code representations even a small number of samples are given. In RQ1, the model has already learned knowledge of downstream tasks in the training phase. It is easier to achieve good performance by continually training on the target domain instead of training from scratch.

We notice that when the data size is extremely small (e.g., 32), the model tends to overfit data. In this situation, zero-shot learning is more preferred.

Different from RQ2 in a cross-language few-shot setting, in this experiment we evaluate the effectiveness of our approach in a monolingual few-shot setting. We train the models with a few samples of Java, Solidity and Go, and evaluate the performance of the tasks in the same language.

Table 3 shows the accuracy of different approaches in three classification tasks. We can observe that Zecoler consistently outperforms baselines in the monolingual few-shot setting. Most of the baseline models simply predict random answers, with an accuracy of around 50%. This indicates that the baseline models cannot learn meaningful code representations with scarce data. Comparatively, Zecoler achieves 75.7% accuracy in average with only 100 data samples. The results suggest that Zecoler learns code representations efficiently in the monolingual few-shot setting.

Figure 6 shows the performance of Zecoler, CodeBERT, and CodeBERTa with different data sizes in the classification task. We can see that Zecoler outperforms the other two baselines under almost all data sizes. Furthermore, as the data size increases, the accuracy of Zecoler grows faster than that of baseline models. This indicates that Zecoler is effective in learning code representations given only a few data samples.

We have also observed that monolingual learning outperforms cross-language learning on small data sizes (e.g., 32 and 100), but achieves similar performance when the data size becomes larger. This is because continuously training on scarce data of a different language can lead to overfitting.

In this experiment, we evaluate the effectiveness of Zecoler in zero-shot generative tasks. Similar to the experiment in RQ1, we initially train a representation model for each task using 5,000 data samples of Java, and then apply the trained model to the target languages (i.e., Solidity and Go) without extra training.

Table 4 and 5 show the accuracy of various models in code summarization and code generation tasks, respectively. We measure the accuracy of code generation using CodeBLEU (codebleu) instead of BLEU except in Solidity which is not supported by CodeBLEU. We can observe that Zecoler consistently outperforms baseline models across most target languages in two tasks. Compared with CodeBERT, Zecoler gains 6.68% and 4.84% improvement in code summarization and 4.35% and 4.33% improvement in code generation in terms of BLEU and Rouge-L.

We also notice that RoBERTa-large and CodeBERTa surpass our approach in the code summarization task of Go. The main reason can be that these models have a bigger model size, which will be further discussed in RQ5.

In this experiment, we inspect the performance of Zecoler under different hyperparameters. We vary the prompt templates and numbers of prompt tokens to search for the optimal prompt template. We also explore the impact to the performance by different source languages and different scales of backbone PLMs.

As shown in Figure 7(a), placing prompt tokens uniformly achieves the best performance compared to other templates. The reason could be that prompts have more influence to nearby tokens. By placing prompts uniformly, every input token can be influenced by sufficient prompts.

As Figure 7(b) shows, the number of prompt tokens is strongly correlated to the performance of representation learning. Fewer prompt tokens are insufficient to steer the PLM to yield meaningful prediction, while large numbers of prompts restrict the input size. The optimal number of prompt tokens is 10 in our experiments. A similar test has been made in generative tasks and the optimal number of prompt tokens is also 10.

This discrepancy could be caused by different model structures between classification and generative tasks. In classification tasks, the models need to encode two separate text inputs and combine them to the fine-tuning header, which makes it difficult for large PLMs to converge in the transfer learning process. However, in generative tasks, PLMs in different sizes take a single text snippet as input and decode the final answer. Hence, larger PLMs, with more common knowledge learned in pre-training phase, have more potential to enhance the downstream tasks.

To demonstrate the effectiveness of our approach, we qualitatively analyze concrete examples of five code intelligence tasks by Zecoler and CodeBERT.

Figure 9(a) shows the results of clone detection for two code snippets in Solidity. They are similar in functionality while being different in key words and structures. For example, “paymentAddress” and “kashout” (highlighted in red) are two equivalent keywords in the two snippets. Because the two words are both domain specific, baseline models such as CodeBERT can hardly detect the clone without prior knowledge. Comparatively, Zecoler successfully detects the clone by reusing prior knowledge from PLMs using prompt learning.

Figure 9(b) shows an example of code search. The query searches for programs that dial with a time-out limit. The target code involves a Go specific API “context.WithTimeout” which can hardly be recognized by CodeBERT. Thanks to the zero-shot learning of Zecoler, our approach successfully recognizes both the domain-specific API and the query, hence gives a correct answer.

Figure 9(c) compares the results by Zecoler and CodeBERT in the method name prediction task of Go. The “panic” in the code is a language-specific API which interrupts the program when an error occurs. As can be seen, CodeBERT can hardly comprehend the function “panic” in zero-shot scenario, while Zecoler can fully understand it and predict the correct method name.

Figure 9(d) presents an example of code summarization. Compared with the summary generated by CodeBERT, the one generated by Zecoler contains more precise keywords such as “Special”. Also, the meaning of “new” in its output is similar to that of ”initializes” in the expected one. In contrast, CodeBERT is not capable of grasping the meaning of the code snippet, and thus the generated summary is far from the expected output.

Figure 9(e) compares the generated code in Go for the query “CompareBS is a barrier session comparator based on seqno”. The Zecoler generates code with correct syntax especially the return and if statements, while many repeated error tokens such as “Sql.” can be found in the code generated by CodeBERT.

These examples demonstrate the superiority of Zecoler in zero-shot code representation learning. The prompts in Zecoler cast the underlying meaning in the PLM to downstream tasks, which help large PLMs capture text and code semantics even without training data.

As the core prerequisite for many code intelligence tasks, learning code representations has been extensively explored in software engineering (codexglue). Broadly, typical approaches in learning code representations can be classified into three categories, including unsupervised for general languages, supervised for specific tasks, and few-shot learning.

The most typical category of work lie in the unsupervised approaches such as code2vec (code2vec), code2seq (code2seq), and InferCode (infercode). Code2vec and code2seq aggregate representations of each path in AST (abstract syntax tree) based on attention. InferCode predicts subtrees automatically identified from the contexts of an AST in a self-supervised manner. These methods directly learn code representation from AST paths. They utilize the word embedding techniques in natural language processing and incorporate them with semantic and syntax information in program source code. The limitation of these methods is the lack of adaptions to downstream tasks. The learned code vectors are fixed and cannot be fine-tuned on downstream tasks. Furthermore, these methods are purely trained on code, thus are unsuitable for NL-PL tasks such as code search.

To improve the performance of downstream tasks, researchers have also resorted to task-oriented supervised learning methods (codenet). For example, for code clone detection task, FangLS0S20 caught the similarity of semantics between two code snippets using a supervised deep learning model, which pays attention to caller-callee relationships and learns the hidden syntactic and semantic features of source codes. ZhangHZWLS21 disentangled the representation of semantic and syntax with AST and GAN (generative adversarial network), then used only semantic representation to detect code clone. For code search task, gu2018deepcs proposed a code representation model named CODEnn to learn semantic representations of code snippets through jointly embedding with comments, HaldarWXH20 designed a multi-perspective cross-lingual neural framework, and Liw20 learned code-query interactions. ZhangCLP21 proposed a hybrid code representation learning approach to resolve program dependence and semantics for predicting method name. YangCZSZ21 learned a unified vector representation of both methods and bug reports for method-level fault localization. ZhouLSD019 constructed a graph neural network to learn semantic representation of code to identify the vulnerable functions. WangL21a proposed AST Graph Attention Block to capture different dependencies in the AST graph for representation learning in code completion. These models are trained for the specific downstream tasks, which achieve good performance but lack generality to support multiple tasks with one single model.

The aforementioned methods require a large scale corpus to train the code representation model. To alleviate this problem, pre-trained programming language models are proposed such as CodeBERT (codebert) and CodeT5 (codet5). It is a fine-tuning based few-shot program learning paradigm: PLMs learn a vast amount of knowledge from large scale unlabelled corpora in the pre-training phase, and achieve state-of-the-art accuracy in the fine-tuning phase with a small amount of labelled task-specific data. This gives PLMs the basic generalization ability to handle a wide range of downstream tasks well. Task adaption through fine-tuning adds extra knowledge of specific tasks to PLMs and improves the performance. However, in this paradigm, the gap between the pre-training phase and the downstream task can be significant: the objectives are different, and for the downstream tasks, we usually need to introduce new parameters.

To the best of our knowledge, our Zecoler is the first zero-shot learning method for code representation. Zecoler follows a prompt-based learning paradigm for task adaption. Prompt learning makes it possible for downstream tasks to take the same format as the pre-training objectives and require no new parameters. By narrowing the gap between the two phases, deploying the PLMs on specific tasks becomes much easier with little training data.

As a promising method for zero-shot learning, a growing number of prompt-based learning approaches (abs210713586) have been proposed in recent years. For example, pet proposed PET which transforms the classification task into an MLM task and uses prompt to elicit knowledge from PLM. But the prompt is manually crafted and hard to select the most suitable words for it. autoprompt proposed AutoPrompt which automatically searches prompt words discretely using gradient signals in the target task. Although discrete searching retains the semantic of prompt, it also cannot find out the most precise prompts for machine models. For solving this problem, instead of using prompt for solving downstream tasks directly, abs-2210-14803 proposed a method that uses prompt for filtering training dataset and trains the model more efficiently. But the quality of dataset is hard to control, which makes the accuracy of downstream tasks low. ArcoVK22 tackled this challenge with combinations of multiple prompts for more robust prompt template. But this way relies on human selection and still can not make sure to find the best prompt. prefixtuning proposed Prefix-Tuning which optimizes a continuous task-specific vector prepended to every layer of the Transformer in PLM and freezes the PLM for saving computation cost. Prefix-Tuning demonstrates superb performance but it only focuses on natural language generative tasks. nomorefinetune empirically evaluated the usage and effect of prompt tuning in code intelligence tasks including defect prediction, code summarization, and code translation. The results show that prompt tuning outperforms fine-tuning in full data, and also shows great potential in low-resource scenarios.

Comparatively, Zecoler optimizes the prompt vectors in continuous space instead of discrete words or human-writing, making the prompt more suitable for PLMs to understand and more efficient for extracting knowledge. Moreover, Zecoler is the first prompt method for zero-shot code representation that can generalize to various code understanding and generative tasks.