aiXcoder-7B: A Lightweight and Effective Large Language Model for Code Processing

The pre-training data of aiXcoder-7B consists of two parts: natural language data and source code data.

Natural Language Data. We collect natural language data from two public datasets: WuDaoCorpora [7] and RefineWeb [8], driven by two key motivations. First, these datasets are highly diverse, covering a wide range of domains and languages. They include a broad spectrum of natural language text from the internet, such as social media conversations, books, and technical papers, and cover two mainstream languages, i.e., English and Chinese. Second, both datasets have been thoroughly cleaned and deduplicated in previous studies, which significantly reduces the preprocessing workload and allows us to focus on processing code data. Finally, we collect 2.8TB of natural language data for pre-training.

Source Code Data. The raw source code data comes from two sources: one is the open-source dataset - The Stack v1.2, and the other is the code data we crawled ourselves.

• •

  The Stack v1.2 [9] is a comprehensive dataset comprising approximately 6TB of permissively licensed source code sourced from public GitHub repositories, spanning 358 programming languages, with notable representation from HTML, JavaScript, Java, and C. This dataset has undergone rigorous cleaning processes to enhance data integrity and prevent inflated performance metrics due to duplicate content. Only repositories with permissive licenses were retained, while non-essential files, such as binaries and those exceeding 1MB, were systematically excluded. The version 1.2 of The Stack has excluded opt-out requests submitted by February 9, 2023, as well as initially flagged malicious files (this exclusion is not exhaustive).

• •

  Our crawled data. Popular repositories we have been crawling from GitHub for the past decade.

In this stage, we clean the collected data by removing invalid or low-quality data. Because the natural language data has already undergone rigorous cleaning, we focus on cleaning the source code data. Our cleaning process comprises two steps: repository-level cleaning and file-level cleaning. Below, we provide a detailed explanation of each step.

Repository-level Cleaning. Our goal is to remove repositories with impressive licenses and low-quality repositories. To achieve this goal, our cleaning is performed in three steps:

• •

  Collecting permissive licenses. We build a list of permissive licenses based on the Blue Oak Council¹¹1https://blueoakcouncil.org/list and previous work [9]. This list includes various permissive licenses with minimal restrictions on software copying, modification, and redistribution. Only repositories with licenses from this list are retained for pre-training.

• •

  Identifying repositories’ licenses. GHArchive provides license information when repository owners explicitly set the code license through the web interface. We first extract each repository’s license from GHArchive. If a license is not listed in GHArchive, we leverage the go-license-detector²²2https://github.com/src-d/go-license-detector to identify the most likely license.

• •

  Removing repositories with impressive licenses. After identifying the licenses, we exclude repositories whose licenses do not appear on the permissive license list.

• •

  Removing low-quality repositories. We score the repositories from different aspects, including the number of stars, the number of git commits, and the number of test files. Then, we sort the repositories in descending order based on their scores and remove the lowest 10%.

File-level Cleaning. Next, we filter out low-quality files in repositories. Specifically, we empirically design some rules to filter out low-quality documents: ❶ Trivial files, including empty files, corrupted files, non-text files, and auto-generated files. ❷ Too long files. Too long files typically contain wordy or repetitive content and are not suitable as training data. If a line in a file exceeds 1000 characters, the total number of lines in the file exceeds 10,000, or the size of the file exceeds 1MB, we consider it a long file.

Previous work [8] has shown that data deduplication can significantly improve the performance of trained models. It is particularly necessary for code data, where code reuse leads to a large amount of duplicate content. Therefore, in this stage, we eliminate duplicate code files within the repositories. Our deduplication process consists of two steps:

• •

  Exact deduplication. We extract file contents, find the files with the same content, and keep only one copy.

• •

  Near deduplication. Exact deduplication is too strict and may cause false positives. Thus, we further perform near deduplication. We compute the MinHash [10] with 256 permutations of all files and use Locality Sensitive Hashing [11] to find clusters of duplicates. We further reduce the clusters by ensuring that each file in the original cluster is similar to at least one other file in the reduced cluster. We consider two files near-duplicate when their Jaccard similarity exceeds 0.85.

In this stage, we use code analysis tools to assess the quality of code data and filter out low-quality code. Low-quality code often contains syntax errors, code defects, vulnerabilities, and misleading models that generate unreliable code. Specifically, we use the following tools to assess the quality of code:

Syntax Parser. Syntax correctness is one of the basic principles that source code should satisfy. We use a public syntax parser - tree-sitter³³3https://tree-sitter.github.io/tree-sitter/ to parse all code files and delete files that fail to parse or time out.

SonarQube. SonarQube⁴⁴4https://www.sonarsource.com/products/sonarqube/ is an open-source tool for the inspection of code quality. It can detect code defects, vulnerabilities, code smells, and technical debt in various programming languages. We use SonarQube to identify problematic code files and delete them.

In this section, we remove the sensitive information in the pre-training data, e.g., texts involving sensitive topics and personally identifiable information (PII). We remove this information in two steps:

Match-based filter. We manually build a list of sensitive words, which covers a broad range of sensitive topics (e.g., politics). Then, we scan all pre-training data and delete the data containing sensitive words. For example, if the frequency of sensitive words in a repository of a user exceeds a certain threshold, all repositories of the user will be deleted.

Model-based filter. Following previous work [4], we use a Named Entity Recognition (NER) model to identify PII in the data. Specifically, we reuse a trained NER model in previous work [4], which can identify six categories of PII, including emails, names, IP addresses, usernames, passwords, and keys. Then, we replace the detected PII entities with the following special tokens: <EMAIL>, <NAME>, <IP_ADDRESS>, <USERNAME>, <PASSWORD>, <KEY>.

aiXcoder-7B is built upon an auto-regressive dense Transformer architecture[12]. aiXcoder-7B consists of 32 Transformer decoder layers, with a hidden state size of 4096 and an intermediate size of 14464. More details are in our open-sourced repository [1]. Our tokenizer is trained with SentencePiece [13] upon 500GB of training data. The vocabulary size is 49,512. We adopt Rotary Positional Encodings (RoPE) [14] to enhance the representation of positional information in sequences, following [4, 2]. RoPE allows for a more flexible encoding of position than absolute position encoding. Additionally, we implement Grouped Query Attention (GQA) [15], which enhances the efficiency of the attention mechanism by grouping queries, allowing for a more scalable attention computation. We maintain a balanced design in our attention heads, with a configuration of 32 query attention heads and 8 key-value attention heads.

Through the pipeline in Section II, we collect extensive code repositories and natural language articles. We randomly shuffle these repositories and articles and iterate through them. If a natural language article is sampled, we process it into training sequences based on the training objectives (Section III-C).

If a code repository is sampled, we design an algorithm for sampling files from the repository, as described in Algorithm 1. The algorithm contains four strategies: sampling based on file content similarity, sampling based on file path similarity, sampling based on inter-file dependencies, and random sampling. The first three strategies simulate common cross-file code completion scenarios, such as code completion augmented by similar code and cross-file API completion, helping aiXcoder-7B better understand and utilize dependencies across files. The fourth strategy, random sampling, is to simulate other potential code completion scenarios. For each repository, the probability of selecting each of the first three strategies is 30%, and the probability of selecting the last strategy is 10%. These sampled files are further converted into training sequences based on the training objectives (Section III-C).

The training objectives of aiXcoder-7B consist of the Next-Token Prediction (NTP) and Structured Fill-In-the-Middle (SFIM), detailed as follows.

Next-Token Prediction (NTP). It is similar to code completion, training models to predict the subsequent token based on the provided context. Given a code file or natural language article $\mathbf{x}=\{x_{0},x_{1},\dots,x_{l}\}$, NTP trains models to predict the next token $x_{i}$ based on previous tokens $\{x_{<i}\}$. The objective is to minimize the following loss function:

Fill-In-the-Middle (FIM) [16]. The motivation behind this training objective is that human developers frequently modify existing code, e.g., inserting new code snippets. Thus, FIM trains models to predict the middle content based on the preceding and following context. Specifically, given a code file or natural language article $\mathbf{x}=\{x_{0},\dots,x_{l}\}$, we randomly select a span of contiguous tokens as the $\mathbf{middle}=\{x_{i},\dots,x_{j}\}$, using the content above the span as the $\mathbf{prefix}=\{x_{0},\dots,x_{i-1}\}$ and the content below as the $\mathbf{suffix}=\{x_{j+1},\dots,x_{l}\}$. We employ two distinct modes to construct the training sequence: PSM (i.e., $[\mathbf{prefix};\mathbf{suffix};\mathbf{middle}]$) or SPM (i.e., $[\mathbf{suffix};\mathbf{prefix};\mathbf{middle}]$). $[;]$ means the concatenation of multiple strings using special tokens. Previous work [3] has found that models work best when PSM and SPM account for 50% each. Thus, we choose the probability of each mode being 50%.

Finally, we feed the training sequence into aiXcoder-7B and minimize the following loss function:

Structured Fill-In-the-Middle (SFIM). As shown in Figure 3, FIM randomly selects spans and trains models to predict incomplete and irregular code snippets (e.g., or i in range(2). However, developers often expect models to complete the current code into a complete snippet, such as a completed code line or loop block, instead of suggesting an incomplete code snippet. To address this, we propose SFIM, which trains aiXcoder-7B to predict complete code snippets, enabling aiXcoder-7B to align with the practical needs of developers. Given a code file, SFIM uses a new strategy for selecting spans: ❶ randomly select a function from the file and parse the function into a syntax tree; ❷ randomly choose a non-root, non-leaf node from the tree and locate the code snippet corresponding to this node; ❸ within this code snippet, randomly select a span, where the start position is randomly determined, but the end position must be the end of a code line. As shown in Figure 3, SFIM selects an If node and mines a relatively complete code snippet (i.e., i%5==0:) as the span.

Subsequently, we follow the FIM and convert the select span into a training sequence in the format of PSM or SPM. Based on preliminary experiments, we set the probability of selecting PSM to 30% and SPM to 70%. We input the training sequence into aiXcoder-7B and minimize the following loss function:

Multi-objective Training. We optimize the above three objectives alternately. Given a code repository, we choose SFIM with a probability of 70% and FIM and NTP with a probability of 15%, respectively. Given a natural language article, we choose FIM and NTP with a probability of 50%, respectively. We determine these probabilities based on previous work [3, 4, 5] and our preliminary experiments.

We leverage Megatron to train aiXcoder-7B. The training process is conducted on 128 A100 40GB GPUs, consuming a total of 1.2 trillion unique tokens. The hyper-parameters used during training are shown in Table I.

Our study aims to answer the following Research Questions (RQs). They evaluate aiXcoder-7B in three code completion tasks, including Natural Language to Code (NL2Code), Fill-In-the-Middle (FIM), and cross-file code completion.

RQ1: How does aiXcoder-7B perform on NL2Code task compared to existing LLMs? NL2Code is the task of completing the source code based on a natural language description or function signature.

RQ2: How does aiXcoder-7B perform on Fill-In-the-Middle task compared to existing LLMs? FIM simulates scenarios where developers modify existing code by predicting the missing middle portion using bidirectional contexts.

RQ3: How does aiXcoder-7B perform on Cross-File Code Completion compared to existing LLMs? This task requires completing code by using relevant context from other files within the current repository.

In the RQs, we apply aiXcoder-7B on 6 benchmarks in total. These benchmarks cover 6 programming languages. To show the superiority of aiXcoder-7B, we also select 10 popular LLMs as baselines for comparison. Then, we report the execution-based and text-based metrics (Section IV-D) of completed programs.

We select 10 popular LLMs for comparison, and they can be divided into two groups:

❶ LLMs with similar sizes. The first group contains six popular LLMs, which have similar sizes to aiXcoder-7B.

• •

  CodeGen2.5-7B [17], released by Salesforce, is a 7B parameter model specialized in code generation and understanding, trained on a diverse set of programming languages.

• •

  CodeGeex2-7B [18], developed by Zhipu AI, is a 7B parameter model designed for code completion and bug fixing, leveraging a large corpus of code data.

• •

  CodeLlama-7B [2], an open-source model by Meta AI, is a 7B parameter architecture fine-tuned on a vast collection of code and natural language data based on Llama2[19].

• •

  CodeShell-7B [20], introduced by Shell AI, is a 7B parameter model focused on shell scripting and code interpretation, trained on a mixture of code and command-line data.

• •

  StarCoder2-7B [4], from BigCode, is a 7B parameter model trained on The Stack v2 dataset, specializing in code understanding and generation across multiple programming languages.

• •

  DeepSeekCoder-7B [3], by DeepSeek AI, is a 7B parameter model trained on a blend of code and natural language data designed for programming tasks.

• •

  Qwen2.5-Coder-7B [21] is a large language model with 7 billion parameters. It builds on the strong Qwen2.5 [22] and continues training on a larger scale of code data, including source code, text-code grounding data, and synthetic data, totaling 5.5 trillion tokens. This leads to significant improvements in code-related tasks.

❷ LLMs with larger sizes. We also select four larger LLMs as baselines to demonstrate the superiority of aiXcoder-7B:

• •

  CodeLlama-13B[2] is an enhanced version of the CodeLlama model with 13B parameters.

• •

  StarCoder-15B[4] is the expanded version of the StarCoder model with 15B parameters, delivering improved accuracy for code synthesis and interpretation.

• •

  StarCoder2-15B[5] is a 15B parameter model upgraded on the original StarCoder, offering refined code generation and more diverse programming languages.

• •

  CodeLlama-34B[2] is the largest variant of the CodeLlama series with 34B parameters.

NL2Code Benchmarks. Following previous studies [3, 23, 24], we select three popular NL2Code benchmarks in our experiments, detailed as follows.

• •

  HumanEval [25] and MBPP [26] consist of 164 and 974 Python programming problems. Each problem includes a function signature, a detailed docstring, and several test cases. LLMs are required to complete the function body based on the signature and docstring. The generated code is checked by executing test cases, being considered correct only if all tests pass.

• •

  MultiPL-E [27] is the multilingual version of HumanEval, covering multiple programming languages, e.g., C++, Java, and JavaScript.

FIM Benchmarks. Code is rarely composed in a straightforward left-to-right sequence. Simulating when a developer modifies existing code, FIM refers to the task of completing missing a middle code snippet leveraging bidirectional contexts.

• •

  Santacoder-data [28] is a popular FIM benchmarks consisting of 4,792 samples. It is built from MultiPL-E [27] and requires LLMs to predict a single line of code based on the preceding and following context.

• •

  FIM-Eval is a large-scale FIM benchmark collected by this paper. We construct FIM-Eval from some real-world repositories, which are excluded from the training data of aiXcoder-7B. We extract 13 types of code snippets from these repositories and randomly mine spans from these code snippets. These 13 types of code snippets encompass common code completion scenarios, including method signatures, method bodies, single-line statements, methods with comments, empty code blocks, specific positions within a method body (top, middle, and bottom), specific control statements (i.e., if statements, for loops, while loops, try statements, and switch-case statements). Finally, we collected 16,140 samples covering four programming languages: C++ (4,080 samples), Java (4,080 samples), Python (3,900 samples), and JavaScript (4,080 samples). FIM-Eval provides a reliable, practical, and diverse evaluation platform for FIM. FIM-Eval has been open-sourced in our repository [1].

Cross-File Code Completion Benchmarks. This task requires LLMs to complete the code based on cross-file context within the same project. Building upon insights from prior research [3, 4], detailed as follows.

• •

  CrossCodeEval [29] covers four popular programming languages: 2,665 Python samples, 2,139 Java samples, 3,356 TypeScript samples, and 1,768 C# samples. Each sample is provided in three formats: no cross-file context, retrieved cross-file context, and retrieval with reference. The completed code snippet is compared using text-based metrics.

We describe the evaluation metrics used in different code completion tasks.

NL2Code. NL2Code benchmarks provide test cases for evaluation. Thus, we execute test cases to check the correctness of the generated code and report Pass@$k$ [25]. Specifically, we generate $n\geq k$ code snippets per testing sample, count the number of correct code snippets $c\leq n$ that pass all test cases, and calculate the Pass@$k$:

FIM and cross-file code completion. We consider the LLMs’ completions as predictions and the human-written completions as references. We compare the predictions to references and compute the following metric:

• •

  BLEU [30] measures the $n$-gram similarity between predictions and references. $n$ is empirically set to 4.

• •

  CodeBLEU [31] is a variant of BLEU for code. It considers not only the $n$-gram similarity but also the syntax and data flow similarity.

• •

  Exact Match (EM) evaluates the percentage of cases where the prediction exactly matches the reference, providing a strict measure of how often LLMs produce correct code without deviations.

• •

  Edit Similarity (ES) measures the similarity between the prediction and the reference based on the number of edits required to transform one into the other, typically using metrics like Levenshtein distance [32].

Following recent work on LLMs [3, 5], we use greedy decoding and report Pass@$1$. Table II shows the results of different LLMs on NL2Code benchmarks. From Table II, we draw the following observations:

• •

  Compared to LLMs of similar sizes, our aiXcoder-7B achieves the current best results, outperforming the top-performing model DeepSeekCoder-7B by an average of 9.8%. Moreover, it significantly surpasses CodeGen2.5-7B with a 31% absolute advantage.

• •

  aiXcoder-7B even surpasses four larger LLMs (e.g., StarCoder2-15B and CodeLlama-34B), achieving a lead of 13.1% over CodeLlama-34B, which is nearly five times larger, and 13.7% over StarCoder2-15B on average.

• •

  Across languages like Java, Python, C++, and JavaScript, our aiXcoder-7B shows strong performance. It surpasses DeepSeekCoder-7B by 16.1% in JavaScript and exceeds by 5.5% in Python.

Generally, FIM closely mirrors how developers modify existing code, making it an ideal method for evaluating models in real-world programming scenarios.

Based on the experimental findings outlined in Table III, aiXcoder-7B demonstrates the highest overall performance on SantaCoder-data, achieving the best results in Python, JavaScript, and Java among the models tested.

Table V shows the average generation performance on FIM-Eval. Figure 4 shows the performance of LLMs in predicting different types of code. Based on the results, we obtain the following observations:

• •

  In real-world programming, aiXcoder-7B performs well in FIM. When evaluated on Java, C++, and JavaScript in FIM-Eval, aiXcoder-7B surpasses DeepSeekCoder-7B by an average of 5.2, 6.7, and 6.4 in FIM metrics for these three languages, highlighting its multilingual versatility. It is highest in C++, exceeding StarCoder2-7B by 11.8.

• •

  aiXcoder-7B offers no clear edge over DeepSeekCoder-7B in Python, likely due to lower training data proportion. When calculating CodeBLEU in FIM-Eval, aiXcoder-7B’s score of 63.0 is slightly lower than DeepSeekCoder-7B’s score of 63.4. In several aspects, such as method body top/mid and if statement, it falls behind by up to 10%, indicating the need for a better understanding of method initiations and conditional branches. Moreover, in the SantaCoder-data benchmark, aiXcoder-7B’s 83.0% EM of Java is 5.1% lower than the best score of 88.1%. This will be rectified by boosting Python and Java data in training.

Another important capability of LLMs is the ability to understand code context across files, as developers often need to consider information from other files within the current project when writing code [33, 34]. In Table IV, we fix the context length for all LLMs at 16K and format the input using the PSM pattern in FIM. All LLMs employ the greedy search to generate code.

We design three experimental settings: ❶ Base Setting. As a baseline, LLMs predict the code based solely on the current file without cross-fire context. ❷ Retrieval BM25. Based on the current file context in the base settings, it additionally uses BM25 to match repository code fragments. The top 5 matches, capped at 512 tokens, are added to the prompt. ❸ Retrieval w/Ref. In this setting, we use not only the in-file context (as in Retrieval BM25 setting) but also the reference to retrieve the cross-file context. We prepend the retrieved context to the in-file context to construct the prompt for this setting.

The subsequent conclusions can be made:

• •

  Under three experimental settings, aiXcoder-7B performs very well, achieving EM of 30.0 and ES of 70.8 in Python, outperforming CodeLlama-7B by 7.7 and 21.4 in the base setting. In the other two experimental settings with retrieval, aiXcoder-7B has an average EM that is higher than DeepSeekCoder-7B by 3.6 and 4.5 and an average ES that is higher by 4.5 and 8.6.

• •

  The model’s performance varies across different languages. aiXcoder-7B excels in C# and achieves an EM of 61 in the third setting. In other languages, the improvements of aiXcoder-7B slightly decrease. For example, it only achieves an EM of 40.4 in Python. In the future, we will continue to enhance the model’s performance across different languages.

The inference efficiency of code completion models is crucial for enhancing the developer’s experience. While scaling up model size can improve accuracy, it often comes at the cost of inference efficiency. Table VI illustrates the inference efficiency of our aiXcoder-7B (bfloat16) and Qwen2.5-Coder-32B (int4) under identical testing settings on a single A100 GPU. Despite the quantization of Qwen2.5-Coder-32B to int4, which slightly reduces its inference quality, its inference speed and throughput are still significantly lower compared to aiXcoder-7B in bfloat16. For example, Qwen2.5-Coder-32B requires 91.65 ms to generate a completion suggestion, whereas aiXcoder-7B achieves a much faster response time of 29.26 ms—approximately 3 times faster. These results demonstrate that aiXcoder-7B can generate completion suggestions both accurately and efficiently, making it superior for practical applications.

We propose a novel evaluation perspective in FIM, i.e., comparing the code length between human-written reference code and code generated by LLMs. It is essential not only to ensure that the completed code is functionally correct but also that its length is consistent with what a human programmer would produce.

To gain insights into this aspect, we evaluate LLMs’ performance using FIM-Eval (Section IV-C), which includes a variety of scenarios. Additionally, we present the code length ratio, which is calculated as the ratio of the number of tokens in the prediction to the number of tokens in the ground truth code. Based on the experimental results in Table VII below, we observe that existing LLMs tend to over-generate, producing code that is substantially longer than necessary. Too long code will increase the burden on users and reduce maintainability. For example, CodeLlama-7B produces ratios of 2.14 for Java and 3.02 for C++, while StarCoder2-7B reaches even higher ratios, such as 3.62 for C++. In contrast, aiXcoder-7B consistently generates code predictions that are similar in length to human-written code, achieving ratios of 0.97 for Java and 0.87 for Python. We attribute this performance to aiXcoder-7B’s Structured Fill-In-the-Middle (SFIM) training objective, which helps align model outputs with human-written code, resulting in more efficient coding practices.

Based on our practices in aiXcoder-7B, we summarize the following insights to help practitioners train the next generations of LLMs for code.

Scaling up training data can continuously improve the performance of LLMs. Although the scaling law [35] provides a relationship between model size and the amount of training data, we discover that further scaling up training data is necessary. Even if the training loss of LLMs is already small, continually training with new data can still improve the performance of models. Similar phenomena have also been found in other works [36].

Exploiting the relationships between code files during training can enhance the LLMs’ ability to understand cross-file context. In practical applications, LLMs often need to predict code based on cross-file context [29]. Thus, during the training process, we should organize files based on their relationships and train LLMs to understand and utilize the cross-file context. For example, we sample files based on the similarity of their content, which is closer to retrieval-augmented code completion scenarios.

Incorporating the code structures into the training objectives can improve the performance of LLMs. The code is highly structured. However, previous LLMs [3, 2, 4] view the code into plain text, ignoring the underlying code structures. This paper first incorporates code structures into the training objectives of LLMs and proposes SFIM. SFIM constructs code spans based on syntax trees of code and trains LLMs to generate accurate and concise code. The results in Section V show the effectiveness of our SFIM. This inspires practitioners to explore new training objectives to model valuable code structures, e.g., control flows and data flows.

We summarize two main threats to this paper.

Data Leakage. A critical threat when training LLMs is the potential inclusion of evaluation data within the training set, which can undermine the reliability of evaluation outcomes. To address this threat, we exclude any data related to our evaluation datasets during data collection. Additionally, the FIM-Eval dataset we constructed and used in our experiments was further emphasized to ensure its independence from the training data. While we cannot guarantee the absence of data leakage in other models due to lack of access, our benchmarks demonstrate that aiXcoder-7B outperforms them reliably.

The selection of hyper-parameters. Another threat to the validity of our study lies in the selection of hyperparameters and rules used during the training of aiXcoder-7B, including model architecture hyperparameters, thresholds for data cleaning, data deduplication parameters, code quality assessment criteria, and sensitive information removal strategies. We selected these based on our preliminary experiments and prior empirical knowledge. We did not conduct a comprehensive hyperparameter search due to the substantial computational costs involved in pre-training, potentially resulting in suboptimal configurations. However, this does not affect our contributions, as future improvements in hyperparameter optimization or heuristic rules can be easily integrated into our framework.