CodeTransOcean: A Comprehensive Multilingual Benchmark
for Code Translation

This dataset contains 30,419 program samples covering eight popular programming languages, namely, C, C++, C#, Java, Python, Go, PHP, and Visual Basic. Table LABEL:table:Multilingual shows the statistics of each language pair. Note that XLCoST Zhu et al. (2022a) is the only existing multilingual code translation dataset. Compared to XLCoST, MultilingualTrans is advantageous in more balanced data distribution across various programming languages, practicality of language pairs, and data quality. For example, the real-world requirement for translating Java into JavaScript as in XLCoST is quite limited. As to data quality, our MultilingualTrans originates from a programming chrestomathy website, with all data already reviewed and verified by the website.

The NicheTrans dataset contains 236,468 program samples, covering code translation pairs from thirty-seven niche programming languages, including Ada, COBOL, Pascal, Perl, Erlang, Fortran, Scala, Julia and others, to the eight popular ones. Table 7 shows statistics of each niche language. Although many studies have highlighted the practical necessity of code translation for modernizing niche programming languages Chen et al. (2018); Zhu et al. (2022b); Rozière et al. (2020), our NicheTrans dataset is the first dataset for code translation between these niche languages and popular ones. We believe this dataset will not only facilitate modernization of outdated programming languages more effectively, but also augment and evaluate generalizability of neural models.

The LLMTrans dataset aims to provide a benchmark for evaluating the performance of LLMs on code translation. The dataset translates seven popular programming languages to Python, totaling 350 program samples. We compile and test these samples and record the execution results. Based on this dataset, we design and implement an automated pipeline, AutoTransExecuter⁸⁸8AutoTransExecuter only supports translation from any source language to Python. We discuss it in Limitations., automatically using LLMs to conduct code translation, execution, debugging, and calculating the success rate. This dataset and the automated pipeline ease investigation of the actual debugging success rate of LLMs on code translation and effectively measure the practical usability of LLMs. Details of the LLMTrans dataset are in Table 1.

The widespread applications of deep learning (DL) has spawned emergence of various DL frameworks, such as PyTorch, TensorFlow, MXNet, and Paddle. However, there are significant differences in syntax and dependency libraries between different frameworks, severely impeding reusability of projects⁹⁹9https://www.assemblyai.com/blog/pytorch-vs-tensorflow-in-2023/. Moreover, studies illustrate significant disparities in energy consumption and economic costs during training and inference between various frameworks Georgiou et al. (2022). Selecting an appropriate DL framework for green AI has become paramount in an era of large models Ananthaswamy (2023). Code reusability and energy-economic efficiency in DL have emerged as critical considerations for both research and practical engineering implementation. Converting code between different DL frameworks is challenging, mainly due to differences between frameworks, code complexity, structural inconsistencies, and cross-platform compatibility (more details are in Appendix A.3). Existing cross-DL-framework adaptive technologies such as the ONNX¹⁰¹⁰10https://onnx.ai/ model conversion protocol require both parties to import and export based on agreed data formats or to convert only the final model through the computation graphs. These technologies have obvious limitations. In contrast, we propose a Cross-DL-framework Translation task for code migration between different DL frameworks through code translation (Appendix A.4). Compared to existing cross-framework adaptive technologies, Cross-DL-framework Translation achieves re-implementation under multiple DL frameworks through an automated process, which not only generates highly readable code and enables secondary development, but also provides developers with flexibility on combining advantages of multiple frameworks.

We construct the DLTrans dataset for Cross-DL-framework Translation, including four deep learning frameworks and spanning twelve directions. To the best of our knowledge, our work is the first to define the cross-DL-framework translation task and construct a corresponding dataset. We create two subsets of different granularities based on the collected code, namely, coarse-grained at the program level and fine-grained at the function or class level. Each code pair comprises code that shares the same functionality but is written in different popular DL frameworks, including PyTorch, TensorFlow, MXNet, and Paddle. The coarse-grained and fine-grained datasets have 408 and 3,270 samples, respectively. In this work, we only experiment on the coarse-grained subset.

In this work, we use pre-trained CodeT5+ Wang et al. (2023b)¹¹¹¹11We will conduct evaluations of a broader selection of models on our datasets in future work, including LLaMA Touvron et al. (2023), WizardCoder Luo et al. (2023), etc. as the backbone based on its superior performance on code understanding and generation evaluations reported in Wang et al. (2023b). We use the MultilingualTrans dataset to investigate four multilingual modeling strategies based on data sharing in the source or target language or both, namely, One-to-One, One-to-Many, Many-to-One, and Many-to-Many, with One-to-One as the baseline. Details of the four strategies are in Appendix A.5. To understand the strengths and weaknesses of the four strategies, we compare their average performance on all language pairs and focus on low-resource and high-resource pairs. Since the CodeBLEU metric Ren et al. (2020) does not cover all eight languages in MultilingualTrans, we use BLEU to measure translation accuracy for the four strategies. Then, we establish baselines for the DLTrans and NicheTrans datasets.

We rank the resource richness of the eight programming languages in MultilingualTrans in descending order based on their amounts in the CodeT5+ pre-training data, as Java, PHP, C, C#, Python, C++, and Go (Visual Basic is not covered by the CodeT5+ pre-training data). Based on this ranking, we consider Visual Basic, C++, and Go as low-resource languages and Java, PHP and C as high-resource languages.

Detailed experimental results are shown in Table 9 in Appendix. For All language pairs, the performance of the four strategies is ranked as One-to-Many > Many-to-Many > Many-to-One > One-to-One. (1) Under One-to-Many strategy, the model encoder can provide more comprehensive information for source language translation due to its ability to absorb more source language features, thereby improving generalizability of the model. (2) Many-to-Many can be considered as expanding the One-to-Many strategy by employing a greater volume of non-source language data for training. Since the encoder must be attuned to the features of various languages simultaneously under Many-to-Many, parameter sharing may potentially undermine the performance. (3) Many-to-One helps the model to learn from a broader range of data than the baseline. Specific patterns or expressions in diverse source languages assist the model in more precisely comprehending how to translate into the target language. The shared semantic representations across different source languages allow the model to implement effective transfer learning strategies. Furthermore, increase in training samples enables the model to optimize the loss function more stably. These results are consistent with previous findings on multilingual modeling for natural language translation Aharoni et al. (2019): Many-to-Many models, trained across multiple target languages instead of just one target language, can function effectively as a regularization strategy for Many-to-One, thereby reducing the possibility of over-matching.

For High-resource and Low-resource languages, as shown in Table LABEL:tab:high_low_resource, the ranking of the four strategies is the same as for All, but there is notable difference in their adaptability across languages of varying resource scales. High-resource languages can take advantage more effectively from the shared information across multiple source languages; whereas, low-resource languages are relatively less equipped to handle the additional uncertainty and noise introduced by shared parameters, and thus often have to rely on a larger volume of source language data to optimize their benefits.

Results from the Many-to-Many strategy on DLTrans and NicheTrans datasets are shown in Tables LABEL:tab:multilingual_modeling_dl and LABEL:tab:multilingual_modeling_niche. The experimental results suggest that significant improvements in translation accuracy can be achieved by swapping the source and target languages in the training set to facilitate data augmentation and training a bidirectional model.

Notably, prior studies on multilingual neural machine translation often overlook the comparison between One-to-Many and other strategies. Nevertheless, One-to-Many demonstrates superiority over the One-to-One baseline across all our experiments. Overall, our results strongly recommend a targeted multilingual modeling strategy for code translation, as it not only can translate multiple language pairs with a single model, but also achieves better and more stable accuracy than baselines.

Prior works show that prompts can influence the performance of ChatGPT Zhong et al. (2023); Peng et al. (2023); Jiao et al. (2023). We set an initial prompt “Translate [SL] to [TL]:[SC].” as the baseline, where [SL] and [TL] denote the source language and the target language respectively and [SC] denotes the source code. We also add “Do not return anything other than the translated code.” for each prompting strategy to require ChatGPT to return only code in order to ease code execution. We design three prompt variants. Details of the experimental settings and prompt variants are in Appendix A.6. We also investigate the effect of hyperparameters on code translation performance.

As shown in Table 3, implementing role assignments, clarifying usage, and polite inquiry in prompts all degrade the performance compared to the baseline prompt. These results show that the baseline with the most straightforward prompt produces the best performance, possibly because it provides clear, short, and unambiguous instructions for the task to the model. More intricate prompting strategies may introduce noise and confuse ChatGPT. The performance of polite inquiry prompt is comparable to but still worse than the baseline performance. We speculate that the improvement from polite inquiries in prior studies Akın (2023) may stem from their explicit and comprehensive formulations which make it easier for the model to understand the task requirements. We also observe in Table 3 that same as prior findings, BLEU and CodeBLEU have no obvious positive correlations with the debugging success rate (DSR@0). Since the reference target code exhibits the same functionality as the source language code but their execution results could differ slightly, EM also does not correlate with DSR@0. Therefore, in subsequent experiments, we only report DSR@0. We also evaluate the CodeT5+_220M model on LLMTrans with the Many-to-Many strategy and find that DSR@0 is 0, suggesting that CodeT5+_220M Zero-shot is unable to generate executable translation results.

ChatGPT selects the token with the highest probability during generation. The hyperparameter temperature influences the randomness of the generated text, while top_p controls the range of vocabulary considered during generation. Higher temperature or top_p could increase diversity in the generated results from ChatGPT. However, as shown in Table 11 in Appendix, independently varying temperature or top_p does not notably change the performance of ChatGPT; hence for the other ChatGPT experiments, we set both temperature and top_p as 0 to ensure stability an reproducibility.

We explore a Divide-and-Conquer strategy, which segments the source language code into snippets (e.g., functions and sub-functions), translate each snippet independently, then merge their outputs as the final result. As shown in Table 3, Divide-and-Conquer significantly degrades the performance. We hypothesize that lack of the global context in Divide-and-Conquer could prevent ChatGPT from considering the overall structure and variable configurations of the code for translation.

Since ChatGPT has shown preliminary capability in error detection and correction during code generation Shinn et al. (2023); Chen et al. (2023b); Kim et al. (2023); Nair et al. (2023); Madaan et al. (2023), we use ChatGPT to perform multiple rounds of self-debugging and investigate the impact on DSR. Specifically, ChatGPT first translates the source language code into the target language (which is Python as in our AutoTransExecuter) and then attempts to execute the translated code. If the execution passes and executing the translated code exhibits the same functionality as the source code, it is regarded as a successful execution. Otherwise, feedback from the compiler will be also fed to ChatGPT for the next round of translation, and this process is repeated until reaching a pre-defined number $K$ of debugging rounds. The whole process is shown in Table A.6 in Appendix. As shown in Table LABEL:table:chatgpt_debug, DSR improves significantly with multiple rounds of self-debugging. The first self-debugging improves DSR by 3% absolutely. Each subsequent round of self-debugging brings further gain but DSR begins to plateau after the second debugging round. This suggests that ChatGPT has limitations in its capacity to rectify errors after multiple debugging cycles, which is consistent with human behaviors.

In-context learning Brown et al. (2020) allows the model to learn from input examples, enabling it to understand and manage each new task. This method has been validated as an effective strategy for enhancing the performance of model inference Peng et al. (2023); Liu et al. (2023a). Therefore, we explore one-shot learning for ChatGPT on code translation. We investigate three one-shot learning sample selection strategies. Descriptions of the strategies and the corresponding prompts are in Appendix A.7.

Table 4 shows that all three One-shot learning strategies effectively improve DSR@0 of ChatGPT over the Zero-shot baseline. The Experiment#2 strategy (provided contextual example has both same source and target languages as the original task) achieves the best performance, yielding 1.72% absolute gain in DSR@0, with Experiment #1 (example has the same target language but different source language) and #3 (example has different source and target languages) following closely with 1.14% and 0.29% absolute gains, respectively. These results show that One-shot learning entirely tailored to the translation requirements is most effective in boosting code translation performance for ChatGPT. The results corroborate previous findings in natural language translation Peng et al. (2023) that the performance of ChatGPT is sensitive to the provided contextual example in One-shot learning.

Chain-of-Thought (CoT) allows the model to simulate an orderly and structured way of thinking by sorting out the thinking process. It helps guide the model to output the final answer step by step Wei et al. (2022); Peng et al. (2023); Kojima et al. (2022). For code translation, we investigate four CoT strategies. Detailed descriptions and translation prompts for each strategy are in Appendix A.8. As shown in Table 4, CoT degrades executability of the translated code. In Experiment #2, DSR@0 even declines by 6% absolutely. We study the translation results of ChatGPT and find that when CoT strategies are applied, the model tends to translate the source code line by line, neglecting compatibility issues between libraries and functions in different languages. CoT also compromises the global planning ability of the model. These observations are consistent with the findings in  Peng et al. (2023) that CoT may lead to word-by-word translations of natural language, thereby degrading the translation quality.

To address the limitations of existing evaluation metrics and our AutoTransExecuter, we propose another novel code translation evaluation metric fuzzy execution using LLMs in Section Limitations, inspired by recent progress in using LLMs as evaluation metrics for NLP tasks. Our preliminary studies evaluates the performance of ChatGPT for predicting whether a given code can be executed or not, and if executable, also for predicting the executed output. Experimental results show that using ChatGPT for fuzzy execution is not yet practical and demands future research.