Unified Abstract Syntax Tree Representation Learning for Cross-Language Program Classification

As illustrated in Figure 1, this model firstly takes different programming source codes as input and then parses them into ASTs. Afterwards, it performs path embedding and graph embedding using unified vocabulary on ASTs. Then the path embedded vector and the graph embedded vector are fed into different sub-networks to capture code features respectively. Finally, it fuses separate learned features and conducts the classification task.

Specifically, it mainly consists of five parts: Unified Vocabulary, Sequence-based AST Network, Graph-based AST Network, Unified AST Feature Fusion, and Cross-language Program Classification. We explain the details of each part in the following subsections.

Since different programming languages have different coding rules and coding characteristics, there are certain differences in the text representation of different languages. Figure 2 shows ASTs generated by tree-sitter³³3http://tree-sitter.github.io/tree-sitter/ parsing the function of adding two integers implemented by three programming languages (Java/C++/Python). The reason why we choose tree-sitter for syntactic analysis of the source code is that it supports 40 programming languages, and also provides a Python API which is easy to use. It can be found in Figure 2 that there are some differences in the node names of ASTs. The root node is called ”program” in the Java AST, ”translation unit” in C++ AST, and ”module” in Python AST. Actually, these three terminologies all represent the same semantics, that is, the coding unit, which generally means the code file or the program. In order to reduce the difference in node names between different languages, we propose a mechanism called ”Unified Vocabulary”, which is mainly used to normalize AST node names in different languages, such as the ”coding unit” mentioned above is unified into ”unit”. Specifically, we use ”unit” to replace ”program, translation unit, module” uniformly, which will reduce the difference between different coding languages. Furthermore, ”block” in Java, ”compound statement” in C++ and ”block” in Python are essentially a code block, so ”block” is used instead. The nodes of the same color in Figure 2 represent the same meaning, so they all will be processed into the same node name uniformly.

We have considered all similar but different expressions of node names in different languages. The unified vocabulary mechanism will unify AST node names generated by different programming languages, alleviating the differentiation caused by different coding characteristics. Thus, the embedded vector generated using the unified vocabulary can learn the feature of various programming languages and thereby tackle the problem of cross-language programming classification.

The foundation of cross-language program classification is to learn semantic and syntactic code features of different programming languages. As for the extraction of the syntactic structure information of the code, we propose a Sequence-based AST network (SAST for short). First, we perform a pre-order traversal of the unified AST, and the obtained path sequence can be regarded as a flattened presentation of the AST. The path sequence contains the global information of the source code, and also shows the syntactic structure characteristics of the source code to a certain extent. In order to extract the dependencies between the nodes within the sequence, we use the self-attention structure (Vaswani et al., 2017). The self-attention mechanism is a powerful mechanism in the transformer structure, which is very effective in extracting internal relationships and can alleviate the problem of long-distance dependence. The calculation formula is as follows:

where the three matrices $Q\in\mathbb{R}^{l\times d}$, $K\in\mathbb{R}^{l\times d}$, and $V\in\mathbb{R}^{l\times d}$ are initialized and generated according to the embedded path sequence vector, and these three matrices are equal in the self-attention mechanism, which could reduce the parameters of the model and can train faster. $d$ is the embedding dimension of the path sequence. $l$ is the length of input path. Dot product is calculated between $Q$ and $K$. $d_{k}\in\mathbb{R}^{d}$ is the dimension of input vector. And $Attention(Q,K,V)$ is the calculated attention score.

In writing code, the context of the code statement often reflects its intent. For example, it needs to declare a variable before using it in C++. The above-mentioned self-attention mechanism has captured the internal relationship of the code embedded vector. So in addition to extracting the internal dependencies of the input source code, it also needs to capture the context dependency of the source code. Therefore, the Bidirectional Long Short-Term Memory (Bi-LSTM) (Graves and Schmidhuber, 2005) is introduced here. The Bi-LSTM could learn features of the input data from two directions, so it can infer the current information from the context of the code. In our proposed SAST network, Bi-LSTM is used to comprehensively consider all available input path information in the context to extract semantic and logic features of the source code. Specifically, the hidden state of the LSTM at each position $t$ of the input path is computed as:

where $\sigma$ is the sigmoid function, $tanh$ is the hyperbolic function, $x_{t}\in\mathbb{R}^{d}$ presents the data at position $t$ of the input path sequence after self-attention, $c_{t}$ presents the hidden unit state of $x_{t}$, and $h_{t}\in\mathbb{R}^{h}$ represents the hidden unit state of the learning layer, which is the final extracted code features. $W_{i},W_{f},W_{o}\in\mathbb{R}^{h\times d}$ are the trainable weight matrices. $\odot$ is the element-wise matrix multiplication operator.

After that, we concatenate the hidden state $\mathop{h_{t}}\limits^{\rightarrow}\in\mathbb{R}^{h}$ learned by the forward LSTM and the hidden state $\mathop{h_{t}}\limits^{\leftarrow}\in\mathbb{R}^{h}$ learned by the backward LSTM to obtain the ${h_{SAST}}\in\mathbb{R}^{2\times h}$, which contains the context features of the code.

Sequence-based AST network has learned the global structure and syntactic features of the code from the path sequence, but the path sequence is a flattened representation, which leads to ignoring some tree-like structure information of the code. We find that except for language-specific library files or package files, the logic of different programming languages is generally the same when writing specific functions. As shown in the Figure 2, the parts enclosed by the dashed lines of the same color all have the same meaning (i.e., the parts outlined by the green dotted line all represent ”return a+b” statement in 3 programming languages). In addition, the parts enclosed by the dashed lines of the same color are also highly consistent in the AST shape. In order to learn the local tree features of the AST, it is easy to think of using GCN to extract the local information of the code, so we propose the Graph-based AST network (GAST for short). We regard the AST structure as a special graph and perform convolution operation on it to extract code semantic features. GCN can aggregate and learn features of neighbor nodes of each node in the unified AST, so that it can capture the local features of the AST, that is, the local semantics of the code.

Figure 3 illustrates the AST of ”return a+b” statement and a two-layer convolution operation on it. Specifically, the ”return statement” node will aggregate the information of its first-order neighbors (”unit, binary expression left, and binary expression right”) with the 1-hop aggregation, which can extract the direct relationship between code statements. As the number of hops increases, the ”return statement” node can also indirectly aggregate the information of their second-order neighbors(”body, $identifier_{1}$, $identifier_{2}$”). Therefore, the ”return statement” node could learn the local structural and semantic features of its neighbors. The right of Figure 3 is the adjacency matrix of the left AST, and it is worth noting that $\widetilde{A}$ adds an identity matrix to its adjacency matrix, indicating that the node could also learn the feature of itself. The graph convolution operation is defined as follows:

where ${H_{i}}^{(l+1)}$ is the feature of node $i$ in the layer ${(l+1)}$, ${H_{j}}^{(l)}$ is the feature of all neighbor nodes of node $i$ (including itself) in the layer $l$; $N$ is the number of all neighbors of node $i$; $\widetilde{A}\in\mathbb{R}^{N\times N}$ is the adjacency matrix $A$ of node $i$ added with the identity matrix; $D\in\mathbb{R}^{N\times N}$ is the degree matrix of $\widetilde{A}$; $W^{(l)}\in\mathbb{R}^{d_{in}\times d_{out}}$ is the trainable weight matrix in the layer $l$.

The aforementioned SAST has captured the global structure and logical characteristics of the code, and GAST has extracted the local structural and semantic feature of the code. In order to comprehensively consider the global and local code semantic feature, a fusion mechanism is further needed. We realize the enhancement of dimensional features through vector concatenation, which can be described as:

where $h_{code}\in\mathbb{R}^{2\times h+d_{out}}$ represents the feature vector after the unified AST feature fusion, $h_{SAST}\in\mathbb{R}^{2\times h}$ represents the global structural feature learned from the flattened sequence, and $h_{GAST}\in\mathbb{R}^{d_{out}}$ represents the local semantic feature learned from the graph-like AST.

After that, we obtain the feature vector which includes the global and local semantic features of the input code, and then we perform a fully connected layer for linear dimensional transformation, and eventually the probability $p_{i}$ is the output through the SoftMax layer. We use the Cross Entropy (De Boer et al., 2005) as our loss function and adopt Adam optimizer (Jiang and Su, 2009) to minimize it. The loss is calculated as follows:

where $k$ is the number of program categories, and $y$ is the label of different programs (if the category is i, then $y_{i}=1$,else $y_{i}=0$), $p_{i}$ is the output after the SoftMax layer.

Our motivation is to verify whether adopting the unified vocabulary and the unified AST feature fusion could improve the performance of our approach, and how those two mechanisms affect the performance of UAST. Taking the above concerns into account, we raise the following Research Questions (RQs):

RQ1: How effective is the proposed UAST compared with the other baselines?

RQ1 is intended to investigate whether the proposed UAST outperforms the other state-of-the-art baselines. There are many other works also explore code representation learning semantics for cross-language program classification, and we select representative ones as baselines and will be described in Section 3.3.

RQ2: How does the unified vocabulary affect the performance of our proposed UAST ?

RQ2 aims to explore whether unified vocabulary could reduce the difference between different programming languages and thus improve the performance of our proposed method. To analyze the impact of unified vocabulary on model effectiveness, we verify the impact of using the unified vocabulary or not on the overall performance through ablation experiments.

RQ3: How does the unified AST feature fusion affect the performance of our proposed UAST?

RQ3 is put forward to evaluate the impact of the unified AST feature fusion. Since the two sub-networks (SAST and GAST) mentioned in Section 2.3 and 2.4 could extract global syntactic features and local semantic structure features respectively, so it is a need to explore an effect of the fusion mechanism on the learned features of two sub-networks. Therefore, we conduct ablation experiments to verify the impact of using the unified AST feature fusion mechanism on the overall performance.

RQ4: How do different parameter settings affect the performance of our proposed UAST?

RQ4 is to examine the impact of UAST’s own parameters on performance. UAST contains two important parameters which could affect the model performance substantially. One is the length of the path obtained by pre-order traversal of the AST, the other is the layer of GCN which determines how many layers of neighbors’ information can be aggregated in the GAST. We set different parameters to find the most suitable parameters which make the best performance.

We use two datasets to evaluate the performance of program classification models. The first dataset inherits from the Bi-TBCNN by Bui et al. (Bui et al., 2019), since the dataset contains two programming languages (Java and C++), we call this dataset Dataset JC. To evaluate our model performance on more programming languages, we collect a dataset from Leetcode which contains 20000 problem solutions of five different programming languages, and we call this dataset Dataset Leetcode. The following is a detailed introduction to these two datasets.

• •

  JC: This dataset includes 10 different categories of programs crawled by Bui et al. (Bui et al., 2019) from GitHub. It contains 5822 Java files and 7019 C++ files. The code files for each category implement the same function, so they are codes that are semantically similar to each other. The dataset is divided into three parts: training set (3), validation set (1), and testing set (1). The specific information of the dataset split is shown in the Table 1.

• •

  Leetcode: We crawl 50 different categories of programs from Leetcode, each of which contains 400 solutions of different programming languages, with a total of 20000 files. The codes under the same problem are all semantically similar to each other. The dataset includes five different programming languages: C, C++, Java, Python, and JavaScript. We use NICAD⁴⁴4http://www.txl.ca/txl-nicaddownload.html to filter out the duplicate code. Considering that each code file contains some salient information (the function name of the solution to each problem written in different programming language is the same, for example, the function name of codes to the problem of ”finding the median of two positive-order arrays” are all called “findMedianSortedArrays” ), so we use ”XXX” instead of all function names to each problem and the user-defined function names will not be replaced. Besides, our dataset can also be used as a benchmark dataset for cross-language program classification task. Since the number of problem solutions in different programming languages is unbalanced, we divide it according to the number of different programming languages, and the ratio of division is: training(3): validation(1): testing(1). The specific division information of the dataset is shown in the Table 1.

We compare UAST with the following state-of-the-art baselines for the cross-language code learning.

• •

  CodeBERT (Feng et al., 2020): It is a variant of BERT (Devlin et al., 2019) and it uses RTD (Replaced Token Detection) for pre-training and adopts deep bi-transformer components, so it finally generates the code features that can integrate context information, which can effectively extract the features of the input sequence. And it learns a pre-trained model for 6 programming languages, so it can be used for cross-language program classification.

• •

  Infercode (Bui et al., 2021): This method applies the self-supervised learning ideas in natural language processing to the AST of the code, and trains the code representation by predicting the automatic context sub-trees of the AST. Because it has been trained on multiple programming languages, it can extract the characteristics of the different programming languages and can be used to classify cross-language programs.

We implement our network in Pytorch. For hardware devices, all experiments are run on a 10-core 3.70GHz Intel(R) Core(TM) i9-10900X CPU and NVIDIA GeForce RTX 3090 GPU server. To ensure the fairness of the comparative experiments, we use the same training set for training. We tune the parameters on the validation set and the specific parameter settings are as follows: the epoch of UAST is 5, the batch size is 64, we adopt Adam (Kingma and Ba, 2015) as the training optimizer, and the learning rate is set to 0.001. As for SAST, the length of the path sequence is unified to 700 for Dataset JC and 200 for Dataset Leetcode (specifically, if it is insufficient, it is padded with 0; if it is exceeded, it is directly truncated). Besides, the embedding dimension of the path is 200, the number of units of the Bi-LSTM is 64, the number of layers of Bi-LSTM is 2, and the dropout rate of Bi-LSTM is set to 0.5; the dropout rate of self-attention is 0.2, and the number of heads of the self-attention layer is 4. As for GAST, the size of the adjacency matrix of the graph is unified to [400, 400], and the number of layers of GCN is 2.

For multi-classification tasks, it is common to use public evaluation indicators (Precision, Recall, F1-score, Accuracy) to evaluate the model performance (Zhang et al., 2019). In order to verify the effect of our model, we use those four evaluation metrics. Detailed definition of those metrics as follows:

where $\overline{TP}$ (True Positive) is the weighted average of the number of samples that are correctly predicted as positive examples for each class; $\overline{FN}$ (False Negative) is the weighted average of the number of samples that are incorrectly predicted as negative examples per class; $\overline{FP}$ (False Positive) is the weighted average of the number of samples that are incorrectly predicted as positive examples per class; $\overline{TN}$ (True Negative) is the weighted average of the number of samples that are correctly predicted as negative examples per class.

In order to answer RQ1, we compare UAST with mentioned algorithms in Section 3.3. Table 2 and Table 3 report the results on two datasets. It can be found from Table 2 that UAST achieves a Recall of 0.9611, a Precision of 0.9631, a F1-score of 0.9617 and an Accuracy of 0.9626, which outperforms other baselines. Meanwhile, the result on Dataset Leetcode also outperforms other baselines, indicating that our UAST is effective for cross-language program classification.

Among them, Infercode learns the characteristics of the sub-tree of the AST through self-supervision and it uses a cross-language corpus for training, so it can be used for cross-language programming classification. However, the extracted feature vector is only 100 dimensions, which cannot fully reflect the characteristics of some long codes, making the final performance worse than the other neural network models. As a powerful pre-trained model, CodeBERT’s bi-transformer structure can effectively extract the context information of the code, so its performance can reach 91% on Dataset JC. However, it learns the feature of code by token embedding, so it cannot reflect the semantic characteristics of the code. UAST comprehensively considers global and local code structure semantic information, and also the unified vocabulary in it can reduce the difference in different programming languages, the performance is better than other baselines.

Furthermore, we conduct experiments on two datasets of different sizes and language types. And the performance of our proposed UAST in both datasets outperforms other baselines, which not only shows the efficiency of our model but also shows that our proposed model has a strong generalization ability.

Result 1: Our proposed UAST significantly outperforms other state-of-the-art baselines in terms of Precision, Recall, F1-score, and Accuracy.

As mentioned in Section 2.2, we propose the unified vocabulary mechanism to reduce the differences between different programming languages. So in order to answer RQ2, we conduct an ablation experiment to explore the impact of the unified vocabulary on model performance. The results are shown in the Table LABEL:tab:ablation_uv.⁵⁵5Method with -V indicates that the unified vocabulary is not used.

According to Table LABEL:tab:ablation_uv, we find that using the unified vocabulary outperforms those who do not use it by 0.37% - 3.32% in terms of Recall, by 0.37% - 3.86% in terms of Precision, by 0.42% - 3.26% in terms of F1-score, and by 0.30% - 2.86% in terms of Accuracy on both Dataset JC and Leetcode, which indicates that the unified vocabulary indeed reduces the feature gap between different programming languages and thus improves the performance of cross-language program classification.

Since different programming languages have their own unique coding features, the terms obtained by the parser in parsing different programming languages are different even if they have the same meaning, so the AST nodes will show different node names. If we directly generate a vocabulary for all the node names of the AST, then each language will have a certain difference in its corresponding AST, which will enlarge the gap of code features. So the use of the unified vocabulary will reduce the difference in terms obtained by code parsing, and further reduce the feature learned by the proposed network, so the performance for cross-language program classification tasks has been improved.

Furthermore, we find that in addition to improving the performance of all three networks (SAST, GAST, and UAST), the use of the unified vocabulary in the SAST network has the most pronounced effect. Due to the fact that the SAST network is designed to extract the features of a traversal path sequence of the AST, and the initial embedded vector input into the network is determined by vocabulary mapping, so the unified vocabulary has the most direct impact on the SAST network which leads to the notable performance. As for the GAST network, the adjacency matrix is not constructed based on the unified vocabulary, but only on the structure of the AST. And the feature matrix is composed of the one-hot vector of the node name, which is related to the unified vocabulary, so the overall impact of the unified vocabulary on the GAST is relatively insignificant compared to SAST.

Given all those results discussed above, the unified vocabulary improves the performance to a certain extent, confirming the effectiveness of the unified vocabulary for the cross-language program classification.

Result 2: The unified vocabulary does improve model performance to a certain extent, so leveraging the unified vocabulary to generate the embedded vector is a good choice for cross-language classification tasks.

Table LABEL:tab:ablation_fusion shows that the performance after the unified AST feature fusion is 1.06% - 4.86% higher than that before fusion on Dataset JC, and 0.66% - 12.58% higher than that before fusion on Dataset Leetcode, indicating that the unified AST feature fusion could further enhance the effectiveness of our proposed model.

In addition, we find that the performance of SAST is not good as the performance of GAST on both datasets. This result implies that the feature extracted from the flattened AST sequence structure is less useful than the feature extracted from the AST graph-like structure. The main reason is that the sequence obtained by traversing the AST has fewer structural characteristics of the branch statements, cause it only includes the overall process of the source code. While the adjacency matrix in the graph-like AST can capture the associated relationship with neighbor nodes, which is very effective for some structures (i.e., for-loop structure, if-condition structure, etc.), the performance of SAST is not as good as that of GAST.

In order to further analyze the reasons why unified AST feature fusion works better, we perform a reasonable analysis as follows. The two code features extracted by the unified AST neural network (SAST and GAST) contain global features and local features of the code respectively. If we only consider one of the features, then the features of the other aspect will be ignored, so fusing the two hidden vectors can strengthen the features of the corresponding dimensions, which results in better performance. Furthermore, the SAST focuses on global information extraction, and the GAST pays attention to the capture of local information. Therefore, two code features are spliced and fused after the unified AST neural network (SAST, GAST), which can comprehensively consider global and local information and learn the semantic structure characteristics of the code from multiple aspects.

Given all those factors stated above, the unified AST feature fusion mechanism can indeed improve the performance of the network as a whole.

Result 3: The unified AST feature fusion mechanism leverages both global structural information and local semantic information for code representation, which can significantly help improve model performance.

In our proposed UAST, the length of the path sequence in SAST and the layers of GCN in GAST are two important parameters that could affect the performance of the model. The length of the path sequence determines how much code content is fed into the neural network. Due to the unbalanced distribution of the length of the source code in the dataset, setting the appropriate length is of particular importance. The number of layers of GCN in GAST, namely hop, represents the range that each node can aggregate information from neighbor nodes. As the number of hops increases, each node can aggregate a larger range of information from neighboring nodes, thereby focusing on a larger range of local semantic information of the code. To answer RQ4, we perform experiments with different parameter settings, and the experimental results are shown in Figure 4 and Figure 5.

For the length of the path sequence, we explore the effect of different lengths on performance by setting different length values (100-1000). Figure 4 shows that the UAST achieves the best performance when the length is set to 700 on Dataset JC and 200 on Dataset Leetcode in terms of Recall, Precision, F1-score and Accuracy. As the length increases, the effect of the model will not be improved, but it will increase the complexity of the model. Besides, Bi-LSTM cannot effectively handle long sequences of inputs, which is one of the possible reasons for the poor performance. When the length is reduced, the performance of the model will also decrease. The main reason is that reducing the length of the input will cut out some useful information, so that the neural network cannot effectively capture some key feature of the code.

Notably, we have statistics on the length of AST path for both datasets. As shown in the Table LABEL:tab:statistic, we find that 80% of the AST path sequences are within 726 in length on the Dataset JC, while 80% of the AST path sequences are within 221 in length on the Dataset Leetcode. Therefore, it is most suitable to set the sequence input to 700 and 200 respectively on Dataset JC and Leetcode, because long input will cause some short codes to be filled with 0, and short input will lose some key information. Thus, when training on different datasets, it is a need to have statistics on the length of AST path and choose the most appropriate length before training.

For the layers of GCN, we explore the effect of different layers on performance by setting different layers of GCN (1,2,3). It can be seen from Table 5 that the model performs best when the number of layers of GCN is set to 2. The first-order neighbors indicate a smaller neighbor range, resulting in a smaller captured tree structure, so the performance is not as good as two GCN layers stacking. A larger number of layers represents a wider range of neighbors, and the experimental results are even worse than that of one layer. Our initial inference is that some specific structures are only displayed in the range of two layers when writing the code. In addition, the increase in the number of layers could make the model more complex and lead to overfitting.

Result 4: For the choice of path length, it is better to have statistics on dataset AST path before training and choose the most suitable value. While setting the layer of GCN to 2 is beneficial to model effectiveness.

In order to capture the characteristics of the code, current research is mainly to learn the code by exploring different code Intermediate Representations (IRs). Token represents code lexical information, Abstract Syntax Tree (AST) is a representation form that includes code structure and grammatical information. Graph-based IRs such as Control Flow Graph (CFG) and Data Flow Graph (DFG) contain the control flow and data flow characteristics of the code. Different representation forms include different features from different views of the code. In recent years, many excellent studies based on those IRs have been carried out (Azcona et al., 2019; Mou et al., 2016; Feng et al., 2020; Ben-Nun et al., 2018). It is currently the mainstream method of program classification to distinguish codes of different categories (functions) by capturing code characteristics.

For the intermediate representation of the code, we have a variety of options, token, AST or graph-based forms (such as CFG, DFG). For the token, it only contains the information of the lexical level of the code, and the program classification is based on the function implemented by the code (i.e., the semantic level), so the token is not the best choice. As for graph-based representations, although some control information they contain can describe the semantics of the code, it is not easy to generate the graph of multiple languages, and the cost of training a pure graph model is high. While AST is relatively easy to generate for most programming languages. And as a tree structure, AST is easier to traverse. So we finally decide to use AST as our code representation.

The essential difficulty of cross-language program classification is to learn the semantic features of codes implemented in different programming languages. This model achieves such excellent performance on cross-language program classification mainly due to two mechanisms (unified vocabulary and unified AST feature fusion) in it. The unified vocabulary reduces the differences between different programming languages, making it more effective for cross-language classification. The fusion mechanism fuses the learned code features of global and local information to capture the semantics of the code leading the classification performance better.

There are two types of threats to this study, and accordingly, we have made efforts to mitigate those threats to validity.

Internal threat: The threats to the internal validity of this study may result from the dataset. Since we collect the data from Leetcode, where the function names of the solutions to the problems are the same. If they explicitly appear in the code, the label will be exposed. So we substitute all the function name of solutions to eliminate this threat. Meanwhile, the dataset may be some duplicate codes, therefore, we deduplicate the similar codes using NICAD tools to reduce this threat.

External threat: In order to rule out that our proposed method is only effective on our Dataset Leetcode, we use two datasets for experiments, which could reduce the threat of accidental results to a certain extent. The results outperform other baselines using the same two datasets, illustrating our model has good generalization ability.

Program classification can be regarded as a high-level abstraction of code. In all source code mining tasks, program classification lays the foundation for various tasks related to source code understanding. Program classification can be applied to many scenarios in the field of software engineering, such as code clone detection (Bellon et al., 2007; Börstler, 1995), bug fixing (Peters et al., 2017), code smell classification (Fontana and Zanoni, 2017), defect classification (Wang et al., 2016), program understanding (Alon et al., 2019), etc. The earliest research on program classification dates back to the last century. (Clark and Darlington, 1980; Börstler, 1995; Jiang and Su, 2009; Taherkhani et al., 2011) used features (such as token, component name, counts of statements, code metrics, etc) to classify programs. Since these studies are classified based on artificially defined rules and extract features at the surface level of the code, their performances are not too prominent, but it starts the beginning of exploration for program classification.

Machine learning has been shown to yield promising results for classification. Ugurel et al. (Ugurel et al., 2002) used Support Vector Machine (SVM) (Drucker et al., 1997) to classify the code’s token set, and Ma et al. (Ma et al., 2018) used SVM, Decision Tree (Quinlan, 1986), and Bayesian Network (Friedman et al., 1997) to classify the code based on the code’s token sequence to determine which types software artifacts are produced by various open source projects at different levels of granularity. Shimonaka et al. (Shimonaka et al., 2016) used four machine algorithms to construct a learning model for the syntactic information of the code, which is used to identify the source code to determine whether the code is automatically generated code.

With the rising trend of deep learning, more and more researches begin to combine neural networks to learn the characteristics of the code. Mou et al. (Mou et al., 2016) proposed a tree-based convolutional network, the kernel of which can be used to capture the structural information of the source code, and performed well in the task of classifying programs. Zhang et al. (Zhang et al., 2019) divided the AST into small sub-ASTs, encoded those sub-ASTs into vectors by capturing the lexical and syntactic features, and then used the Bi-RNN model to generate the code vector representation. This algorithm has achieved excellent results in the source code classification task. Barchi et al. (Barchi et al., 2021) explored the use of Convolutional Neural Networks (CNN) (Kalchbrenner et al., 2014) to analyze program source code and proved that the CNN model can be successfully applied to source code classification. Compared with the most advanced methods, this method provided higher accuracy and less learning time.

The emergence of deep neural networks technology has brought new solutions to the field of software engineering, and at the same time, an increasing amount of attention has been devoted to the learning of code representation. Code representation forms are used to analyze the original meaning of the source code at different levels. For example, the token represents the lexical information of the code, while the Abstract Syntax Tree (AST) represents the code structure and grammatical information obtained by the syntax analyzer, and the Data Control Flow (DFG) and the Control Flow Graph (CFG) represent the code data flow and control flow information. Also, there are other intermediate representations (IRs) to represent the code, and different representation forms combined with neural network to learn code semantics is the direction that researchers are exploring currently.

Harer et al. (Harer et al., 2018) used Word2Vec (Mikolov et al., 2013) tool to generate the initial embedded vector for the C/C++ token, and then used the TextCNN (Zhang and Wallace, 2015) model to learn the features of the vector for software vulnerability detection. Azcona et al. (Azcona et al., 2019) analyzed the Python code submitted by students by embedding the token into a vector, and then extracted the effective features of the student to analyze the progress and performance of the student. The token contains the lexical information of the code, but ignores the structural features and grammatical logic information of the code.

Mou et al. (Mou et al., 2016) proposed a convolutional neural network named TBCNN based on AST. They designed a sub-tree kernel to slide on the AST to extract the structure information of the tree, and also used dynamic pooling to deal with the number of sub-trees of the AST. Uri Alon et al. (Alon et al., 2019) proposed a code embedding algorithm called code2vec, which gives an attention score to each path in the AST, so that it could extract the syntactic information of the entire source code while dividing sentences of different degrees of importance. Wang et al. (Wang et al., 2020) constructed a graph called Flow-Augmented Abstract Syntax Tree (FA-AST) to represent source code, which contains control flow and data flow information. Then they used Graph Neural Networks (GNN) (Scarselli et al., 2008) to capture the feature of FA-AST.

Ben-Nun et al. (Ben-Nun et al., 2018) tried to learn code semantics based on the IR of the code. They converted IR into context flow graph (XFG), which contains the data flow and control flow of the code, and then used neural networks to learn the code features from XFG embedding. Wei et al. (Wei et al., 2020) generated a type dependency graph from source code, which links type variables with logical constraints as well as name and usage information, and then used GNN to extract the feature of this kind of IR.

Cross-language learning is widely used in machine translation in the field of NLP (Huang et al., 2013; Conneau and Lample, 2019), and cross-language code learning is still in its infancy. Nguyen et al. (Nguyen et al., 2014) proposed a tool called semSMT based on SMT to migrate Java programs to C#, and it operated at all three lexical, syntactic, and semantic levels to extract code features. Bui et al. (Bui et al., 2019) proposed a bilateral dependency neural networks to learn the features of different languages, and then used Siamese network for similarity detection. Ye et al. (Ye et al., 2020) proposed the MISIM (Machine Inferred Code Similarity System) model, in which the Context-Aware Semantic Structure (CASS) can capture the code’s contextual semantics to describe the intent of the code, and CASS could also learn the language-independent representation through manual configuration for cross-language detection. The Infercode proposed by Bui et al. (Bui et al., 2021) implemented self-supervised learning by predicting the automatic context sub-trees of the AST, in which vectors are generated by training multiple languages, and it performed well on multiple tasks.The TPTrans proposed by Peng et al. (Peng et al., 2021) performed feature learning for different programming languages, and incorporated the feature information of the code tree structure into the transformer structure using position embedding and embedding. This model performed well for code summarization tasks.