Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Models

ECT posits that code-switching occurs only when it does not violate the syntactic rules of either language Poplack (1980). Winata et al. (2019) applies ECT by simplifying sentences in terms of a linear grammatical structure and allowing lexical substitution on non-crossing alignments between parallel sentences (e.g., lexical substitution between ”sentence” and ”vaaky” in Figure 1). Denoting $L_{1}$ as the source language and $L_{2}$ as the target language, given a sentence in $L_{1}$ comprising an array of words $u_{t}={a_{1},a_{2},...,a_{m}}$ and a corresponding sentence in $L_{2}$ comprising an array of words $v_{t}={b_{1},b_{2},...,b_{m}}$, the alignment between $a_{i}$ and $b_{i}$ does not satisfy the constraint if there exists a pair $a_{j}$ and $b_{j}$ such that ($a_{i}<a_{j}$ and $b_{i}>b_{j}$) or ($a_{i}>a_{j}$ and $b_{i}<b_{j}$). If a switch occurs at this point, it alters the grammatical order in both languages, rendering the switch unacceptable. During the generation step, we permit any switches that do not violate this constraint.

While adhering to the core principles of Equivalence Theory Poplack (1980), our approach applies and builds on the relaxed version of ECT introduced by Winata et al. (2019) (Section 2.1) to the task of code-switched sentence generation. This relaxation allows for greater flexibility in identifying potential switching points, accommodating the complexities of real-world code-switching patterns while maintaining grammatical coherence. Our implementation expands on the linear grammatical structure and non-crossing alignment criteria outlined in Section 2.1, introducing additional flexibility to capture a broader range of code-switching phenomena. In the following section, we outline our approach to implementing the relaxed ECT for identifying switching points, developing code-switched sentence generation techniques, and establishing a comprehensive evaluation framework.

Building on the theoretical foundation established earlier, our implementation of ECT to generate code-switched sentence comprises of three main stages: 1) obtaining translations, 2) bitext alignment, and 3) code-switched sentence generation.

We evaluate our approach using three different LLMs: Aya23 Aryabumi et al. (2024), Llama3 8B and Llama3.1 8B Dubey et al. (2024). Aryabumi et al. (2024) introduce Aya 23 8B, an open-weight large language model specifically designed for multilingual tasks. It is trained on a diverse corpus of languages, making it particularly suitable for code-switching applications. Dubey et al. (2024) develop Llama3 8B, a compact yet powerful language model from the Llama3 series. With 8 billion parameters, it offers a balance between computational efficiency and performance, making it ideal for exploring code-switching in resource-constrained settings. Llama3.1 8B is an improved version of Llama3 8B. This model features refined training techniques and an updated dataset, incorporating the latest advancements in language modeling and potentially offering enhanced performance in multilingual and code-switching tasks. All experiments are run on a single NVIDIA L40 GPU with 48GB of memory.

We consider two directions of code-switched generation in our experiments: English to Code-Switched and Indic Language (Hindi, Tamil, Malayalam) to Code-Switched. These directions apply to each language pair in our study, allowing us to examine code-switching patterns initiated from both languages.

We do not use ECT-identified switching points. Instead, we simply instruct the model to create a code-switched version of the input sentence without specifying particular words to include. As another comparison, we We use switching points identified with ECT using the gold (human) translations from the parallel datasets (HinGE for Hindi, Samanantar for Tamil and Malayalam), we called it Human ECT. The model is given specific words from these human translations to include in the code-switched output.

Our experiments utilize three parallel corpora, each corresponding to a distinct language pair, as detailed in Table 1. For the human translations employed in our Human ECT method, we rely on the parallel sentences available in these datasets. In contrast, for the LLM translations used in EZSwitch, we generate translations using the Llama3 8B model.

In our evaluation, we assess the effectiveness of various automatic metrics in capturing human judgment scores of fluency and accuracy of the code-switched text. Specifically, we experiment with COMET (Rei et al., 2020), permuting the reference language to gauge sensitivity across different reference settings. Additionally, we employ GPT-4o-mini as an evaluator to explore leveraging large language models (LLMs) for assessing code-switching, given the subjectivity inherent in human evaluations. This approach allows us to compare traditional automatic metrics with LLM-based evaluation in terms of alignment with human preferences.

We perform human evaluations of the code-switched sentences with native speakers who are bilingual and use code-switching in their daily communications. We adopt a setup inspired by Kuwanto et al. (2024) and reformulate the evaluation to focus on rating sentences rather than expressing a preference. Human evaluators, contractually employed by DeccanAI (previously, SoulAI) are asked to score the accuracy and fluency of the code-switched sentences. DeccanAI ethically recruits human evaluators in India, and evaluates them to determine their English and native language proficiency via their crowdsourcing platform. They also onboard evaluators, train them with the annotation guidelines, and obtain the annotation scores through their annotation platform.

This on 150 sample of inputs from dataset described in section 3.4 with 18 different generation settings. Totaling 2700 sentence to rate for each language. In total, we conduct 24,300 human evaluations in total. For each code-switched sentence, we ask 3 unique evaluators to score the accuracy and fluency of the sentence on a discrete scale from 1 (lowest) to 3 (highest). While evaluating, the evaluators can see (1) the English sentence, (2) the sentence in the respective native language, and (3) the LLM generated code-switched sentence.

We evaluate the entire dataset described in Table 1, each of the input pairs yielded 18 different generations from the different method and model. We use GPT-4o-mini, as traditional metrics often fail to align with human judgment for code-switched text. GPT-4o-mini was used to assess both Accuracy (GPT4o_a) and Fluency (GPT4o_f) for all model and method combinations. The results are split into translations from English input and Indic input to allow for a detailed comparison of each method’s performance across different language directions. Word Replacement (WR) refers to the method proposed in Winata et al. (2019), which we use for comparison against our proposed method.

We performed human evaluations on a sample of the entire dataset, using a random sample of 150 input sentences per Indic language. Given that we tested 3 methods, 3 models, and 2 translation directions (i.e., English $\leftrightarrow$ Indic), this resulted in a total of 2,700 ratings per Indic language. Table 3 shows the result of each of three primary methods: Baseline, Human ECT, and EZSwitch, evaluated across three models: Aya23, Llama3, and Llama3.1. We use human evaluation of Accuracy and Fluency ratings as our primary evaluation criteria, alongside automatic metrics such as COMET and GPT4-based scores (denoted as GPT4o_a for Accuracy and GPT4o_f for Fluency). The table is divided into two main sections based on the direction of translation: From English Input and From Indic Input, allowing us to compare the performance across different language directions. Each method and model combination is evaluated to identify which setup achieves the highest fluency and accuracy as judged by human evaluators. Additionally, COMET is included as a traditional automatic metric for comparison, while GPT4-based evaluations provide an alternative automated approach to assessing fluency and accuracy using large language models.

This subsection explores the relationship between various automatic evaluation metrics and human judgments for both Accuracy and Fluency in code-switched text generation. Understanding these correlations is crucial for determining the reliability of automatic metrics as proxies for human evaluation. Table 4 presents the Kendall’s Tau correlation scores between human ratings and several automatic metrics, including BLEU, COMET, and BERTScore, along with GPT4-based evaluations. Higher correlation values indicate a stronger alignment with human ratings, suggesting the suitability of a metric for capturing fluency and syntactic coherence in code-switched outputs. This experiment was done on the same 150 subset of input pairs from Section 4.2, which yielded 2,700 samples for each language, and we measure the correlations of these metrics with the average human ratings.

We construct CSPref, a pairwise preference dataset using human ratings to evaluate the performance of different models in code-switched text generation. Each pair consists of two generated code-switched sentences compared based on their human-evaluated accuracy and fluency scores. To further analyze the performance, we split the dataset into easy and hard subsets. The easy subset includes pairs where the difference in human ratings is high (indicating a clear preference for one generated sentence), while the hard subset consists of pairs with minimal differences (indicating ambiguous preferences). Table 5 provides the statistics for the pairwise dataset across three languages: Hindi, Tamil, and Malayalam. We report the total number of pairs, as well as the breakdown into “easy” and “hard” subsets for each language pair.

Our experimental results, as presented in Table 2, demonstrate that even small, open-source models like Llama3 8B can produce high-quality code-switched sentences when guided by linguistic constraints. The ECT-guided generation outperformed the baseline in both fluency and grammatical accuracy, illustrating that linguistic theory can effectively enhance the outputs of language models. Notably, Llama3.1 8B consistently achieved higher accuracy and fluency scores compared to other models, highlighting the importance of the language model’s architecture and parameterization in code-switching generation.

The three generation methods (Baseline, Human ECT, and EZSwitch) offer valuable insights based on human evaluation. EZSwitch outperforms Human ECT for translations from English input, demonstrating better fluency and syntactic coherence. This suggests that EZSwitch excels at handling English-to-Indic code-switching. For from Indic input, human evaluations show that EZSwitch and Human ECT perform similarly, but both underperform compared to the Baseline. This may be due to the abundance of data where Indic languages act as the matrix language and English as the embedded language, making LLMs more adept at handling this type of code-switching, as they already possess knowledge of such patterns. When assessed using automatic metrics, Human ECT performed best across all measures, highlighting its ability to capture linguistic nuances that LLM-generated translations might miss. However, EZSwitch still showed significant improvement over the Baseline, demonstrating that ECT-guided approaches can be applied effectively in low-resource scenarios. These automatic metric results underscore the need for metrics that better reflect the complexities of code-switching.

An unexpected finding was the directional asymmetry in model performance between different translation directions. Specifically, translations from Indic languages to English exhibited significantly higher fluency and accuracy than translations in the reverse direction. This asymmetry may be attributed to the training data distribution, where code-switching from Indic languages to English is more prevalent in real-world usage, making the model more familiar with this direction. Additionally, all annotators being native speakers of Indic languages and L2 learners of English might introduce bias. Their proficiency in their native language allows for more rigorous scrutiny of translations into Indic languages, potentially leading to stricter evaluations of fluency and accuracy. Conversely, translations from Indic languages to English might be rated more leniently, as English is a second language for the annotators, making them less critical of subtle errors. This directional asymmetry could also reflect real-world code-switching patterns that favor English as the matrix language, with Indic languages filling lexical gaps. Future research could explore how this asymmetry relates to linguistic theories, such as the Matrix Language Frame model, and examine the impact of annotators’ bilingual proficiency on these evaluations.

Evaluating code-switching fluency and accuracy is challenging due to the limitations of existing automatic metrics. Based on Table 4, while SentenceBLEU reduces reliance on human-generated references, it falls short in measuring fluency. Embedding-based metrics like BERTScore and COMET, although better at capturing semantic similarity, exhibit weak correlations with human evaluations, around 0.2. In contrast, GPT-4o-mini, when used as an evaluator with structured guidance, achieves a higher correlation (Kendall’s tau of 0.5). This indicates that LLMs can effectively approximate human judgment, albeit with potential stylistic biases. These findings underscore the need for specialized code-switching metrics that integrate syntax, semantics, and context, emphasizing naturalness and linguistic validity over traditional n-gram-based matching. Developing tailored metrics is essential for accurately assessing multilingual text generation, as conventional tools fail to capture the complexities inherent in code-switching.