Bridging the Language Gap: Enhancing Multilingual Prompt-Based Code Generation in LLMs via Zero-Shot Cross-Lingual Transfer

The use of Large Language Models (LLMs) for code generation, such as generating Python programs from problem specifications, has gained substantial interest due to their effectiveness in handling complex language tasks (Zhao et al., 2023; Gao et al., 2023; Austin et al., 2021). This capability has led to the development of innovative applications like GitHub Copilot Yetistiren et al. (2022) and specialized LLMs such as CodeLLaMa Roziere et al. (2023), underscoring the growing importance of this field. Although LLMs are globally prevalent and proficient in processing multilingual inputs, they often exhibit biases against non-English prompts Talat et al. (2022); Choudhury and Deshpande (2021), which is particularly evident in code generation. Figure 1 under Baseline Output illustrates how the quality of generated code diminishes as prompts shift from English to other languages, a disparity linked to the availability of data used in LLM training and fine-tuning. Ensuring LLMs deliver consistent quality across languages is crucial for fostering fair and inclusive code generation, especially as the global programming community is increasingly composed of non-English speakers. Data from coding platforms like The Competitive Programming Hall of Fame¹¹1https://cphof.org/countries highlight the skew towards non-English-speaking regions. As the global population of coders grows, addressing these biases in LLMs is essential to promoting an equitable and accessible environment for developers worldwide.

This paper tries to bridge the language gap in multilingual prompt-based code generation by enhancing LLMs through self-supervised fine-tuning. We first evaluate the performance of LLMs like CodeLLaMa and GPT-4 on English and five non-English languages, using a translated and quality-checked version of the MBPP dataset Austin et al. (2021). Significant disparities in code quality across languages were observed, even when using the same problem statements. Inspired by Shi et al. (2022a) and Qin et al. (2023), who improved LLM performance on multilingual tasks with Chain-of-Thought (CoT), and Awasthi et al. (2022), who used bootstrapping for multilingual data generation, we form strong baselines for code generation with CoT and fine-tuning on bootstrapped multilingual data. However, these approaches showed only marginal and inconsistent improvements.

To address data sparsity and limited multilingual exposure, we propose a novel approach: (a) using a pre-trained multilingual encoder like LASER Artetxe and Schwenk (2019) to encode multilingual inputs into a joint vector space; (b) projecting these embeddings into the LLM’s input space and aligning them through training solely on English data; and (c) employing this LASER$\rightarrow$LLM pipeline at inference for zero-shot cross-lingual processing of non-English inputs. This method familiarizes models with multiple languages without needing additional external training data. Our evaluation shows improved code quality and reduced syntax and logical errors, as illustrated in Figure 1 under Zero-shot Cross-Lingual Inference. Our approach integrates seamlessly as a minor training step that does not require any expensive pretraining or fine tuning, thus offering a promising way to enhance LLMs’ multilingual capabilities in code generation.

The contributions of the paper are as follows:

1. 1.

   We create a novel multilingual test dataset with quality-checked translations and new evaluation metrics.

2. 2.

   We introduce a scalable projection technique by integrating the LASER multilingual encoder with popular open-source LLMs like CodeLLaMa Roziere et al. (2023), CodeGemma Team (2024), and Mistral Jiang et al. (2023) for zero-shot cross-lingual code generation.

3. 3.

   Our evaluation of the approach against Chain-of-Thought (CoT) and fine-tuning with multilingual bootstrapped data, highlights the strengths and limitations of each method.

We will make our code²²2https://github.com/lmd0420/Multilingual_Code_Gen and multilingual evaluation data³³3https://huggingface.co/datasets/Mingda/MBPP-Translated publicly available for academic use.

As AI technology advances, transformer-based LLMs like GPT (OpenAI, 2023), LLaMA (Touvron et al., 2023), Mistral Jiang et al. (2023), and Gemma Team et al. (2024) have become prominent in research and applications. While pre-trained models such as LLaMA2 and Gemma are fine-tuned for code generation, their English-centric training data limits multilingual proficiency (Lai et al., 2023; Akiki et al., 2022). Studies show these models face performance issues with non-English tasks (Shi et al., 2022b; Becker et al., 2023), and human supervision remains crucial for quality (Sarsa et al., 2022).

To address these gaps, Ahuja et al. (2023) developed a multilingual benchmark for evaluating LLMs, revealing performance drops across languages. Tan and Golovneva (2020) and Huang et al. (2023) suggest leveraging transfer learning and cross-lingual prompting to improve multilingual capabilities. Additional methods include language-specific pre-training (Pfeiffer et al., 2022) and consistency regularization for fine-tuning (Zheng et al., 2021).

Optimizing prompts enhances multilingual LLM accuracy (Zhao and Schütze, 2021; Huang et al., 2022), and CoT techniques improve code generation (Ma et al., 2023). Fine-tuning with multilingual synthetic data, including translation and back-translation (Sennrich et al., 2015; Hoang et al., 2018), further refines LLMs, as demonstrated by Li et al. (2023) and Zhang et al. (2024). Contrary to these popular approaches, we take an orthogonal route by using specialized multimodal encoders and lightweight projectors to bridge language gaps in popular LLMs. Our work is inspired by multimodal AI literature, integrating projection techniques for different modalities such as language, vision and speech (Liu et al., 2024; Fathullah et al., 2024; Beyer et al., 2024).

This section details our experimental setup, including the creation of a multilingual benchmark dataset, the models evaluated, and the metrics used. Our focus is on Python code generation from multilingual prompts, though the methods and insights are applicable to other languages and contexts.

Given that language models exhibit emergent capabilities and scale effectively across tasks and languages, efficient prompting and prompt tuning are generally preferred over costly training or fine-tuning that demands extensive data curation. Based on our experimental setup, we highlight the challenges LLMs face with multilingual code generation in conventional settings, providing a detailed analysis of existing models’ performance and their limitations. Throughout this section, we will reference Table 3 for a comprehensive discussion of the results.

Our approach focuses on avoiding the use of in-language training data, which can be costly and impractical. Instead, we utilize an intermediate, lightweight method that relies on abundant English data and the LASER multilingual encoder Artetxe and Schwenk (2019), which provides joint embeddings for over 200 languages. In this setup, the LASER encoder preprocesses and embeds input tokens before passing them to the LLM, which then operates on these embeddings rather than raw input IDs. This method enables efficient language scaling, as similar meanings are represented consistently across languages (e.g., the English token "add" and its Hindi counterpart "JoDaNe" are embedded similarly, as shown in Figure 2 part (b)).

Two key challenges arise with this approach: (A) differing tokenization between the multilingual encoder and the LLM, and (B) the LLM’s unfamiliarity with the multilingual embeddings. To address (A), we use word tokens and extract mean-pooled embeddings from subwords using tokenizers such as NLTK ⁸⁸8https://www.nltk.org for space sparated lanaguge inputs, Jieba ⁹⁹9https://github.com/fxsjy/jieba for Chinese, and Janome¹⁰¹⁰10https://mocobeta.github.io/janome/en/ for Japanese. We then train a projector to align these embeddings. For a given word token, we compute the LLM’s subword embeddings ($\hat{H}_{llm}$) through max pooling, and the multilingual embeddings ($H_{laser}$) from the LASER encoder. The projector, with learnable parameters $\mathbf{W}_{llm}$ and $\mathbf{b}_{llm}$, is defined as:

The model is trained by minimizing the Mean Squared Error (MSE) between $\hat{H}_{llm}$ and $\mathbf{H}_{llm}$:

where $N$ is the number of word tokens. Training utilizes English tokens from the MBPP dataset, which includes 127 examples. We do projector training on a single consumer grade NVIDIA 4060 GPU and training the projector happens in 200 epochs in less than one hour. During inference, tokens are first word-tokenized and embedded using LASER, then projected, and finally input to the LLM for multilingual processing without requiring in-language data. To enhance performance and align with baselines, we also concatenate system prompt embeddings with the original programming prompt embeddings. Notably, LASER embeddings are of size 1024, while LLM embeddings are typically 4096 or larger, necessitating a 4-fold upsampling. We achieve this using two linear projection layers as outlined in the above equations. We use the abbreviation LP to explain this system henceforth.

Table 3 presents the overall performance models and variants discussed in sections 4 and 5. Our observations indicate that across all metrics, our proposed model consistently reduces the performance gap between English and non-English languages, as reflected in the differences and deviations. This improvement is particularly evident when comparing the direct querying setup (Orig.) with our multilingual projector-based variant (LP), where deviations from English are generally smaller. We explore the details of each metric below.

In this paper, we demonstrated the significant potential of Large Language Models to bridge language gaps and promote inclusivity in multilingual prompt-based code generation. While LLMs exhibit promising capabilities across various languages, their performance can be inconsistent, particularly with non-English prompts. Our comprehensive analysis and evaluation using a benchmark dataset revealed both strengths and limitations in multilingual code generation, highlighting areas needing improvement.

We showcased the effectiveness of bootstrapping multilingual training data and fine-tuning LLMs to enhance code generation quality across multiple languages. Our zero-shot cross-lingual transfer approach, utilizing projected embeddings, proved effective, as evidenced by improved ATPR and reduced TotalER values. This method eliminates the need for extensive external multilingual data, maximizing the model’s potential internally. Future work will expand this approach to include more languages, diverse prompt patterns, and programming languages beyond Python. Our findings underscore the importance of advancing these techniques to enhance LLM adaptability and utility for a global audience, stressing the need for ongoing efforts to improve their effectiveness and versatility in diverse linguistic contexts.

A major limitation of this work lies in the reliance on word tokenization and pooled token embeddings, which introduces external dependencies and may not scale effectively to extremely low-resource languages where tokenizers are not readily available. Furthermore, the sequence of projected embeddings from the target language can significantly differ from the canonical English order, potentially hindering the model’s ability to fully leverage these embeddings. This misalignment could contribute to the generation of hallucinatory and erroneous outputs. To address this issue, some degree of fine-tuning of LLMs with denoising objectives may be necessary.

Moreover, our exploration is limited to only five non-English languages, which, while a promising start, is not comprehensive enough to establish the approach as a fully robust multilingual solution. Additionally, our study focuses solely on generating code from scratch and does not cover code-filling scenarios, which is another important aspect that warrants future exploration. Due to resource constraints, the scope of this study has been limited to Python, but expanding the approach to encompass other general-purpose and special-purpose programming languages is essential for broader applicability.

The models we utilized in our study are widely used ones from OpenAI, Google, MistralAI and Meta, and we employed Google Cloud Translator and the MBPP dataset on Hugging Face. All of these resources are publicly accessible; we did not introduce any additional real-world data, thus avoiding the creation of new ethical and privacy issues.

Given we are dealing with black-box Large Language Models as part of this study, there needs to be careful consideration of any potential biases that can be harmful in nature. Although we are focusing on a objective task with little to no opinion sourcing from the models, cultural and racial biases can occur given we are exposing the models to multi-lingual prompts. Since the applications we are focusing on are essentially user-centric in nature, a proper communication protocol should be established that can help clarify potential erratic behaviour of models, especially for low-resource languages. We would also like to share that we employed OpenAI’s ChatGPT-4 system to enhance writing efficiency by generating LaTeX code, ensuring concise sentences, and aiding in error debugging.