MultiTEND: A Multilingual Benchmark for Natural Language to NoSQL Query Translation

Early research on Text-to-SQL primarily focused on meticulously designed rule-based methods, such as those in (Baik et al., 2020; Li and Jagadish, 2014a, b; Quamar et al., 2022; Sen et al., 2020), these methods used predefined rules or semantic parsers to translate NLQs into SQL but were inflexible and inadequate for handling increasingly complex database structures. With the rise of deep learning, the focus of Text-to-SQL research has gradually shifted towards methods that utilize deep neural networks, such as attention mechanisms (Liu et al., 2023) and graph-based encoding strategies (Hui et al., 2022; Li et al., 2023b; Qi et al., 2022; Wang et al., 2020; Xu et al., 2018; Zheng et al., 2022; Yu et al., 2021; Xiang et al., 2023). Alternatively, some approaches treat Text-to-SQL as a sequence-to-sequence problem by using encoder-decoder structured Pre-trained Language Models (PLMs) to generate SQL queries (Cai et al., 2018; Popescu et al., 2022; Qi et al., 2022).

In recent years, large language models (LLMs), which have demonstrated remarkable success across various domains, have also garnered increasing attention in the Text-to-SQL field (Dong et al., 2023; Gan et al., 2021; Gao et al., 2024; Li et al., 2023a; Lin et al., 2020; Pourreza and Rafiei, 2024; Qi et al., 2022; Rubin and Berant, 2021; Scholak et al., 2021). Current literature primarily focuses on two approaches with LLMs: prompt engineering and pretraining/fine-tuning. Prompt engineering methods typically involve using specific reasoning workflows which can be categorized into several reasoning modes, including Chain-of-Thought (CoT) (Wei et al., 2022) and its variants (Pourreza and Rafiei, 2024; Liu and Tan, 2023; Zhang et al., 2024b, 2023), Least-to-Most (Zhou et al., 2023; Gan et al., 2021; Arora et al., 2023), and Decomposition (Khot et al., 2023; Tai et al., 2023; Pourreza and Rafiei, 2024; Wang et al., 2025; Xie et al., 2024). To evaluate Text-to-SQL model performance in practical applications, several large-scale benchmark datasets have been developed and released, including WikiSQL (Zhong et al., 2017), Spider (Yu et al., 2018), KaggleDBQA (Lee et al., 2021), BIRD (Li et al., 2023c), and Bull (Zhang et al., 2024a) etc.

Traditional SQL databases face limitations with large-scale, unstructured, or semi-structured data in the internet and big data era, prompting the rise of NoSQL databases, which provide flexibility, scalability, and high performance in web applications and real-time data analysis Moniruzzaman and Hossain (2013). In the field of databases and NLP, current research primarily focuses on several key areas of NoSQL databases,including achieving scalability in data storage systems within large-scale user environments (Cattell, 2011), ensuring consistency in NoSQL databases (Diogo et al., 2019), addressing multi-tenant NoSQL data storage issues in cloud computing environments, particularly in scenarios involving resource and data sharing (Zeng, 2015), and realizing scalability, elasticity, and autonomy in database management systems (DBMS) within cloud computing environments (Agrawal et al., 2011).

Despite the extensive research on NoSQL across various domains, its accessibility remains a challenge, especially for non-expert users. Although Text-to-NoSQL tasks have been proposed to address this issue, existing NoSQL generation primarily supports English and overlooks the needs of non-English users. To tackle this issue, we introduce the Multilingual Text-to-NoSQL task, which is based on existing Text-to-NoSQL research and not only aims to reduce the barrier for non-expert users to utilize NoSQL databases by automatically converting NLQs into NoSQL queries but also addresses the gap in existing Text-to-NoSQL tasks that mainly support English while neglecting non-English users’ needs. In this task, we also introduce MultiTEND, the largest multilingual benchmark for natural language to NoSQL query generation.

Figure 5 presents detailed statistics of (NLQ, NoSQL) Pairs and distinct NoSQL Queries across different languages in MultiTEND. Figure 4 displays the statistics of NoSQL queries in MultiTEND (covering all six languages). Specifically, figure 4(a) uses a pie chart to illustrate the distribution of different query methods in MultiTEND, while figure 4(b) and figure 4(c) show the counts of stages in the aggregate method and operators in the find method, respectively, with a heatmap included to represent the proportion of queries that use each specific stage or operator relative to the total number of queries employing the corresponding method.

Table 5 conveys detailed statistical information about all databases (covering six languages) in the MultiTEND dataset, which includes a total of 924 databases spanning 105 domains. The top five most represented domains in the dataset are Sport, Customer, School, Shop, and Student. Across all databases, there are a total of 2,082 collections, 197,874 documents, and 35,760 fields. Each database contains an average of 2.25 collections, with values ranging from 1 to 7. The number of documents averages 214.15 per database, spanning from 3 to 13,694. Field counts average 38.7 per database, varying from 7 to 331.

To clarify the challenges posed by multilingual Text-to-NoSQL tasks to existing models, we first fine-tuned the Llama-3.2-1B model for testing and found its accuracy to be very low. Through detailed experiments (as shown in Figure 6), we further analyzed the additional challenges multilingual Text-to-NoSQL tasks pose to existing models. Results show that the common errors in NoSQL query generation from different models are primarily caused or worsened by multilingual issues. Additionally, we compared the performance of SMART Lu et al. (2025), a model specifically designed for Text-to-NoSQL, on a pure English dataset (TEND) and a mixed dataset of six languages (MultiTEND)(See Table refappx.tab:Performence Comparison of baselines on MultiTEND and Table 7). Results show that despite being designed for NoSQL generation, the quality of NoSQL queries generated by SMART significantly drops in multilingual contexts. Based on the experimental results, we analyzed the factors leading to the decline in NoSQL generation quality in multilingual environments, with the findings as follows:

• •

  Finding 1: In the process of identifying entity mentions in NLQ and mapping them to corresponding database fields (i.e., schema linking), significant lexical differences across languages, especially in those with more complex lexical formation rules (such as Japanese hiragana and katakana, Russian Cyrillic characters, and the morphological variations in German and French), impose higher demands on the model’s language understanding capabilities, resulting in a significant drop in mapping accuracy.

• •

  Finding 2: The support for nested documents and array structures in NoSQL requires models to have a stronger understanding of database schemas in multilingual environments to handle complex nested fields or situations requiring array expansion. This makes the already challenging schema linking task even more difficult due to the multilingual context.

• •

  Finding 3: In multilingual contexts, NLQs often exhibit vastly different syntactic structures due to language differences, significantly increasing the difficulty for models to comprehend multilingual questions. This also leads to errors in two critical tasks: mapping intentions to operators (intention mapping task) and mapping intentions to database fields (schema linking task).

Based on these experiments and findings, we categorize the challenges in MultiTEND into Structural Challenge and Lexical Challenge.

We utilized a variety of popular neural network models, LLM-based prompting methods,SLM-based fine-tuning methods and existing Text-to-NoSQL pipelines as baseline models for a comprehensive performance comparison with MultiLink. The baseline models are as follows:

• •

  Zero-shot LLM: The zero-shot prompting approach utilizes the inherent zero-shot learning capabilities of LLM, allowing LLM to produce precise and contextually appropriate responses without the need for prior training or example-driven instructions.

• •

  Few-shot LLM: The few-shot prompting technique serves as a key mechanism for in-context learning (ICL), where a limited set of examples is incorporated into the context to instruct LLM on executing tasks within specialized domains.

• •

  RAG for LLM: Retrieval-Augmented Generation (RAG) technology provides an alternative approach to support LLM in downstream tasks. Unlike direct few-shot prompting, RAG dynamically retrieves relevant examples from a knowledge base based on the model’s input, enriching the context and effectively reducing hallucinations induced by insufficient or ambiguous information.

• •

  Fine-tuned SLM: Fine-tuning is another effective strategy for enhancing the performance of language models in specific downstream tasks, such as predicting NoSQL query generation. We fine tune SLM based on two different approaches (Monolingual Training and Multilingual Training) to compare the quality of NoSQL queries predicted by SLM based on single-target language training data and training data from multiple languages.

• •

  SMART: SMART is the first and currently the only framework in the Text-to-NoSQL domain tackling the task of converting English NLQs to NoSQL queries. With the assistance of SLM and RAG technologies, it constructs four main processes: SLM-based schema prediction, SLM-based query generation, query refinement based on predicted schema and retrieved examples, and execution results-based query optimization.

• •

  MultiLink: MultiLink is the framework proposed in this work, aiming to address the challenges of multilingual Text-to-NoSQL tasks. It constructs three main processes: Intention-aware Multilingual Data Augmentation (MIND), Parallel Multilingual Sketch-Schema Prediction,and Retrieval-Augmented Chain-of-Thought Query Prediction.

We report results using the same metrics as SMART, which include Exact Match (EM) and Execution Accuracy (EX), each with more detailed subdivisions such as Query Stages Match (QSM) and Query Fields Coverage (QFC) under EM, and Execution Fields Match (EFM) and Execution Value Match (EVM) under EX.

Here are detailed descriptions of each metric:

• •

  Exact Match (EM): The purpose of this metric is to evaluates whether the generated query is an exact match to the gold query, considering both its structure and content. It is calculated as:

  $$EM=\frac{N_{em}}{N}$$

  Where $N_{em}$ represents the count of queries fully matching the gold query, and N signifies the total number of queries within the test set. EM serves as a stringent measure of syntactic and semantic alignment.

  • –

    Query Stages Match (QSM): QSM is designed to check if the generated query’s key stages (e.g., match,group, lookup) mirror the gold query in the order and keywords employed. It’s calculated as:

    $$QSM=\frac{N_{qsm}}{N}$$

    where $N_{qsm}$ represents the count of queries with matching stages.

  • –

    Query Fields Coverage (QFC): QFC assesses if the generated query encompasses all the fields present in the gold query, considering both database fields and query-defined fields. It’s defined as:

    $$QFC=\frac{N_{qfc}}{N}$$

    Where $N_{qfc}$ represents the number of queries with complete field coverage.

• •

  Execution Accuracy (EX): This metric evaluates the accuracy of the execution results for the generated query on the database. It is calculated as follows:

  $$EX=\frac{N_{ex}}{N}$$

  where $N_{ex}$ represents the number of queries whose execution results align with those of the gold query. EX serves as the most critical performance metric for evaluating Text-to-NoSQL models.

  • –

    Execution Fields Match (EFM): EFM validates the alignment of field names derived from the execution of the generated query against those obtained from the gold query. It’s defined as:

    $$EFM=\frac{N_{efm}}{N}$$

    where $N_{efm}$ represents the number of queries with matching field names in the results.

  • –

    Execution Value Match (EVM): EVM evaluates the correspondence between the values in the execution results of the generated query and those in the gold query. It is defined as:

    $$EVM=\frac{N_{evm}}{N}$$

    where $N_{evm}$ represents the number of queries with matching values in the results.

As shown in Table 8, models consistently achieve higher performance in English across all metrics, suggesting baseline models are more proficient in handling complex tasks in English. This can be attributed to their extensive training on English data, giving them an inherent advantage over other languages. In contrast, performance in Chinese and Russian often lags behind languages like English, Japanese, French, and German. This disparity may arise not only from limited training data for these languages but also from the unique challenges posed by Chinese character construction, syntax, and the Cyrillic alphabet in Russian, which introduce additional complexities in comprehension and generation.

According to the results shown in Table 8 and Table 9, there are significant differences in the performance of various models on key metrics. Our model performs exceptionally well across all languages and metrics except for EVM, outperforming the second-best baseline model by approximately 5% to 20%. Particularly on the two key metrics for real-world applications, EM and EX, our model maintains an absolute lead with average advantages of 25.54% and 59.12%, respectively, across all languages. In contrast, the Zero-shot LLM method performs the worst across all languages and metrics, particularly on EM and EX, with its average accuracy across six languages trailing MultiLink by nearly 25%. The other three methods achieve average accuracies across the six languages on EM of 15.64% (Fine-tuned), 14.32% (RAG for LLM), and 10.82% (Zero-shot LLM), while on EX, their average accuracies are 47.95% (RAG for LLM), 40.95% (Fine-tuned Llama), and 36.69% (Few-shot LLM). Overall, our model demonstrates significant advantages across all metrics on every language.

Further analysis of Table 9 reveals distinct performance differences between model fine-tuning and direct prompting LLM methods on the MultiTEND test set under cross-domain criteria. Fine-tuned Llama achieved an 81.41% accuracy in EFM, outperforming RAG for LLM by 7.19%, indicating its stronger capability in generating queries that retrieve correct field results. On the other hand, RAG for LLM surpassed Fine-tuned Llama by margins ranging from 3.8% to 8.75% in EX, QSM, and QFC metrics, demonstrating its superior understanding of the mapping relationships between NLQs and NoSQL database fields, as well as a deeper grasp of data operations in NoSQL queries, leading to higher execution accuracy

Among the Multi-Step methods specifically designed for Text-to-NoSQL tasks, the performance gap between SMART and our proposed MultiLink approach is significant. As shown in Table 8, specifically, in terms of performance across every language, MultiLink outperforms SMART, which is tailored for monolingual NoSQL generation, across all metrics except EVM, with margins ranging from 2.18% to 18.78%. This demonstrates that SMART, designed exclusively for English contexts, cannot directly address the challenges posed by multilingual tasks. In contrast, the intention mapping and schema linking methods designed in MultiLink, specifically developed to tackle multilingual generation challenges, effectively overcome these difficulties, enabling it to achieve significantly better performance in multilingual scenarios compared to other baseline models.

Figure 6 illustrates the performance variations of MultiLink across multiple languages under different numbers of retrieved examples for all metrics. The vertical axis represents the model’s accuracy under each metric, and the horizontal axis represents the RAG number ranging from 0 to 10.

Analyzing Figure 6 and observing the changes in the average represented by the red dashed line, we find that as the RAG num increases, MultiLink shows significant improvement on all metrics except QSM, followed by a slow rise and a gradual decline after reaching a certain value. The initial improvement, resulting from the change in RAG num from 0 to 2, indicates that RAG greatly enhances model performance. The subsequent decline in MultiLink’s performance beyond a certain value might be attributed to excessive context length introducing redundant information, which interferes with generation.

Moreover, the model’s performance on the QSM metric declines as the RAG num increases, which may be attributed to the influence of retrieved examples on the model’s decisions regarding NoSQL operations. However, considering the performance on other metrics, the retrieved examples contribute to the model generating more accurate queries before the RAG num reaches 6, with the peak accuracy of model execution occurring at a RAG num of 6.

Table 10 presents a case study comparing various baseline methods and MultiLink in generating NoSQL queries and their execution results. Firstly, we observe that the fine-tuned Llama exhibits a significant lack of understanding regarding database field names and overall structure, as illustrated in Table 10. The model not only fails to accurately match target fields (such as “课程名称”), leading to erroneous query logic and invalid results, but also incorrectly identifies “课程” as an independent collection rather than recognizing it as a field nested within the “科目” collection. Additionally, it lacks the ability to deconstruct nested fields (e.g., using $unwind), rendering it incapable of properly handling array-type nested structures.

On the other hand, RAG for LLM performs well in handling the syntax of complex operations, but it still falls short in understanding field names and matching database schemas, and it fails to accurately incorporate the actual values in the database for retrieval. As shown in Table 10, although the model correctly uses $unwind to deconstruct nested fields, the query still cannot execute correctly because it does not filter based on the actual values in the database.

In comparison, SMART demonstrates a solid grasp of NoSQL syntax and shows a slightly better understanding of user query intentions and database structures compared to methods that directly utilize LLMs through prompting for generation. However, it struggles with handling more complex fields that involve nested relationships. Due to a lack of multilingual knowledge, it often incorrectly maps NLQ to database fields, finding it difficult to distinguish between semantically similar fields in different languages. As illustrated in Table 10, SMART erroneously selected the “课程” collection instead of the “科目” collection and failed to correctly structure the nested fields. In contrast, MultiLink fully and accurately comprehended the query intent even in multilingual contexts, with clear query logic and the ability to generate queries that precisely align with the query intent. This indicates that MultiLink can effectively understand and execute multilingual Text-to-NoSQL tasks and generalizes well to test set examples after learning from the training set.