Boosting the Capabilities of Compact Models in Low-Data Contexts
with Large Language Models and Retrieval-Augmented Generation

The specific task we address in this paper is morphological glossing, a component of the automatic production of interlinear glossed text (IGT). IGT is a richly-annotated data format widely used in linguistics, especially as one product of work documenting and describing endangered languages.

The data format, an example of which appears below, consists of multiple interrelated tiers containing different types of linguistic information. This Uspanteko example is representative of a common IGT configuration, with one tier for the original utterance, one for a morphological segmentation, one for a detailed labeling of the component morphemes, and one line with a translation into a language of wider communication.

\ex\begingl\gla

xqil// \glbx-$\emptyset$-q-il// \glcCOM-A3S-E1P-ver// \glft‘lo vimos’ (‘we saw it’)// \endgl\xe

We focus specifically on the glossline, in which we see a mix of stem translations (e.g. ver (in English, to see) for the Uspanteko stem il) and labels indicating morphosyntactic functions (e.g. COM indicates marking of completive aspect on the verb stem). Glossing is a sequence-to-sequence problem that can be approached in 2 ways. The first approach is to train a model to segment the data and then view it as a token classification task. The second is to view it as a translation problem with uneven input and output lengths, thus requiring an encoder-decoder model that can perform sequence-to-sequence conversion. In this paper, we use the former approach. The IGT task was the focus of a SIGMORPHON 2023 shared task. The task, resources, and previous models are described in detail in Ginn et al. (2023).

We directly use the segmented data in track 2 of the Sigmorphon 2023 glossing shared task to train our compact model. The corrective LLM only sees the segmented output of the compact model and the unsegmented text in the training data.²²2https://sigmorphon.github.io/sharedtasks/2023/

Analyses like those by Conneau et al. (2020) reveal the inadequacies of large language models in capturing the nuances of less-common languages. These studies underline the necessity for specialized models that cater to the unique characteristics of low-resource languages. Furthermore, position papers like Bender and Koller (2020) critically analyze the bias and limitations in LLMs, advocating for more inclusive and adaptable language technologies. Another interesting approach to incorporating linguistic information is described in Tziafas et al. (2023). They directly apply syntactic supervision at the pre-training stage to enhance the model with syntactic awareness. While this approach shows promising results, pre-training again requires large amounts of labeled data which is not available in a low resource setting.

Introduced by Lewis et al. (2021), Retrieval-Augmented Generation (RAG) represents a significant advancement in the field of large language models (LLMs) for enhancing generative tasks. By dynamically retrieving information from knowledge bases during inference, RAG effectively addresses issues such as the generation of factually incorrect content, often referred to as “hallucinations.” The integration of RAG into LLMs has been rapidly adopted, making it a crucial technology for enhancing chatbot capabilities and making LLMs more practical for real-world applications.

Naive RAG is the most basic form of retrieval augmented generation. The retrieve-read framework, which was described by Ma et al. (2023), explains the process of indexing and vectorizing reference documents, retrieving relevant chunks based on vector similarity, and generating outputs based on compound prompts that combine the chunks and the query. The idea of RAG has since been expanded and adapted to several domains. Yan et al. (2024) suggest a corrective RAG (CRAG) approach that incorporates a lightweight retrieval evaluator to test the quality and relevance of the retrieved content. The information is then filtered or accepted in the process of producing the final output. In our paper, we also add a corrective step in a different context. Instead of evaluating the retriever, we make the LLM itself generate confidence scores for each of its predicted outputs.

Other recent work on the IGT task takes various approaches. Ginn et al. (2024b) build a very large, multilingual corpus of IGT and use it to finetune a ByT5 model Xue et al. (2022), achieving good results especially on languages not seen in training. He et al. (2024) train models to extract IGT directly from audio data, and Ginn et al. (2024a) explore the use of in-context examples to teach LLMs to gloss low-resource language data. Using the same dataset we use, they find that LLM performance improves dramatically with targeted selection of examples. Even with no traditional training or fine-tuning, models like Gemini 1.5 Pro, Cohere’s Command R+ and GPT-4o outperform transformer baselines. We do not explore few-shot prompting techniques, but it is possible that the performance of our corrective LLM can be further enhanced in this way.

The Naive Retrieval-Augmented Generation (RAG) approach enhances the performance of two compact token classification models trained for glossing of Uspanteko and Arapaho. The process begins by indexing and vectorizing reference grammar documents using a dense vector representation like BERT embeddings Devlin et al. (2018). Here, we use OpenAI embeddings Achiam et al. (2023).

Let $D={d_{1},d_{2},\ldots d_{n}}$ be the set of $n$ grammar document chunks, where each $d_{i}$ is represented as a dense vector $v_{i}$ in a high-dimensional space $R^{d}$. We experimented with different chunk sizes and chunking strategies and found that a default chunk size of 400 characters with a 50-character overlap on either side provided the best results across 5 trials. We also tried chunking according to headings in the grammar, but this resulted in chunks of uneven sizes that did not optimally aid the retrieval process and contextual inference.

Given an input query $q$, in this case the sentence to be glossed along with the attempted prediction of the compact model, the retriever module computes the cosine similarity between the query embedding $v_{q}$ and each document chunk embedding $v_{i}$:

$$\text{sim}(q,d_{i})=\frac{v_{q}\cdot v_{i}}{\|v_{q}\|\|v_{i}\|}$$

The top-$k$ most similar document chunks $\mathcal{D}_{q}=\{d_{q1},d_{q2},\ldots,d_{qk}\}$ are retrieved based on their cosine similarity scores. These chunks are concatenated with the original input query to form a compound prompt $P$:

$$P=[q;d_{q1};d_{q2};\ldots;d_{qk}]$$

The prompt $P$ is then fed into an LLM, which interprets the linguistic rules and morphological patterns described in the retrieved grammar excerpts $\mathcal{D}_{q}$. It uses this information to identify and correct potential errors in the glossing output $g_{s}$ generated by the smaller token classification model:

$$g_{c}=\text{LLM}(P,g_{s})$$

where $g_{c}$ represents the corrected glossing sequence. To illustrate the correction process, let $f_{s}$ be the function learned by the smaller token classification model that maps the input sentence $x$ to the glossing output $g_{s}$.

The Naive RAG approach learns a corrector function $f_{\text{RAG}}$ that takes the original input $x$, the glossing output $g_{s}$, and the retrieved grammar chunks $\mathcal{D}_{q}$ to produce the corrected output $g_{c}$:

$$g_{c}=f_{\text{RAG}}(x,g_{s},\mathcal{D}_{q})$$

By leveraging the linguistic information retrieved from the grammar documents, the RAG model $f_{\text{RAG}}$ is able to refine the predictions of the base model $f_{s}$ and generate more accurate glossing sequences. The Naive RAG approach thus enables the smaller model to benefit from the vast knowledge captured by the LLM without the need for extensive fine-tuning or additional training data. This is particularly advantageous in low-resource scenarios where labeled data is scarce, as the LLM can provide valuable linguistic insights to guide the glossing process. [Refer to the appendix to see some of these generated explanations.]

In addition to correcting glossing labels, our RAG pipeline also generates explanations justifying the corrections made. We achieve this by prompting the LLM to provide a chain-of-thought reasoning trace that justifies the decision-making process behind the corrections. The LLM is prompted with an instruction $I$ that requests a justification $J$, an explanation of how RAG was used $R$, and a confidence score $C$ for corrected glossing output $g_{c}$:

$$[J,R,C]=\text{LLM}(I,P,g_{s},g_{c})$$

The justification $J$ is a natural language explanation that describes the grammar rules and morphological patterns retrieved from the grammar excerpts $\mathcal{D}_{q}$ and how they informed the corrections made to the glossing sequence. This explanation can be modeled as a set of reasoning steps:

$$J=[r_{1},r_{2},\ldots,r_{m}]$$

where each $r_{i}$ represents a single reason that links the retrieved linguistic information to the specific corrections made in $g_{c}$. This set of $r_{i}$ is not necessarily sequential.

To quantify the model’s confidence in the corrected output, a confidence score $C$ is generated. This score reflects the LLM’s certainty in the accuracy of the final glossing sequence based on the retrieved grammar rules and the original output $g_{s}$. When asked how it assigned confidence scores, this was Claude’s response:

"Confidence scores were assigned based on how closely each word or morpheme matched information provided in the grammar document. Higher scores (closer to 1.0) indicate a strong direct match, while lower scores (closer to 0.5) indicate a more tentative match based on context or inference."

Through these confidence scores and justifications, the RAG pipeline boosts the interpretability of the model’s predictions. The explanations offer insights into the linguistic reasoning behind the corrections, allowing users to understand why certain changes were made. This can be crucially important for documentary linguists who may be reluctant to use NLP tools due to concerns about reliability and a lack of interpretibility.

Due to the size of our grammars, it makes sense to further train our retriever and rank the retrieved content based on its relevance to the query. This is an extension of the previously described Naive RAG approach. The modular RAG approach allows for a more sample-efficient utilization of the available grammar resources. By learning to retrieve and prioritize the most relevant excerpts, the model can focus on the linguistic information that is most beneficial for each specific input, rather than processing the entire grammar at once.

1. 1.

   Initial Retrieval: The process begins by retrieving $k$ context chunks from the grammar based on relevance to the input query.

2. 2.

   Retrieval Module Training: The retrieval module, which is a different instance of the RoBERTa model, is fine-tuned based on the performance of the LLM outputs. The input to the retrieval module consists of all the initially retrieved chunks of context, and the output is a relevance vector $r=[r_{1},r_{2},...,r_{k}]$, where $r_{i}\in[0,1]$ indicates the relevance score for the $i^{th}$ chunk of context.

3. 3.

   Final Context Selection: Out of the $k$ retrieved pieces of context, only the top-$n$ most relevant pieces are selected for use in the final LLM prompt.

Let $f_{s}$ be the token classification model that maps input sentence $x$ to the initial glossing output $g_{s}$. The retriever module $f_{r}$ is trained to select the top-$n$ relevant grammar chunks $D_{q}$ based on the input sentence $x$ and the initial glossing output $g_{s}$:

$$D_{q}=f_{r}(x,g_{s},k,n)$$

where $k$ is the initial number of retrieved chunks and $n$ is the final number of selected chunks $(n\leq k)$.

The selected grammar chunks $D_{q}$ are concatenated with the input sentence $x$ and the initial glossing output $g_{s}$ to form a prompt $P$:

$$P=[x;g_{s};D_{q}]$$

The prompt $P$ is then fed into the LLM to generate the corrected glossing sequence $g_{c}$:

$$g_{c}=\text{LLM}(P)$$

During training, the retriever $f_{r}$ and token classification model $f_{s}$ are jointly optimized using a combined loss function:

$$L=L_{s}(g_{c},g_{t})+\alpha\cdot L_{r}(D_{q},D_{t})$$

where:

• •

  $L_{s}$ is the sequence loss between the corrected glossing sequence $g_{c}$ and the ground truth glossing labels $g_{t}$.

• •

  $L_{r}$ is the retrieval loss that encourages the retriever to select relevant grammar chunks. It is implemented as a ranking loss between the retrieved chunks $D_{q}$ and the chunks that led to the best LLM performance $D_{t}$.

• •

  $\alpha$ is a hyperparameter that controls the weight of the retrieval loss.

By jointly optimizing the retriever and token classification components, the modular RAG approach enables the model to learn to identify the most relevant grammar information and effectively incorporate it into the glossing process. The retriever learns to select excerpts that are most pertinent to the input sentence and initial gloss output.

During inference, the trained retriever $f_{r}$ is used to select the top-$n$ relevant grammar chunks $D_{q}$ for each input sentence $x$ and initial glossing output $g_{s}$. These selected chunks are provided to the LLM to generate the corrected glossing sequence $g_{c}$.

Our expectation is that the best results for this task will be achieved by combining a model that exploits all available training data (the compact transformer or LSTM model) with the analytical power of modern LLMs.

More specifically, we use two baseline glossing models, both of which have been shown to achieve strong performance in the standard setting for the glossing task Ginn et al. (2023). Following this setting, we model the production of IGT as a token classification task rather than as a sequence-to-sequence task. Specifically, we experiment with two baseline models: one using RoBERTa Liu et al. (2019), the second using a Bi-LSTM architecture. By exploiting the contextual information captured by these architectures, we aim to obtain accurate predictions of the morphological labels for each token in the input sentences. These predicted labels are then used as the initial glossing output in the RAG framework.

For the RoBERTa baseline, we use the same setting as the baseline for the IGT shared task, as described in Ginn et al. (2023). The input sentences are tokenized and encoded using the RoBERTa tokenizer and encoder. The encoded representations are then passed through a linear classification layer to predict the morphological labels for each token.

For the Bi-LSTM model, input sentences are first tokenized and converted into word embeddings. These embeddings are then fed into the Bi-LSTM layer to obtain the contextualized token representations. A linear classification layer is applied on top of the Bi-LSTM outputs to predict the morphological labels for each token. Both models are trained using cross-entropy loss and optimized using the Adam optimizer. We use an adaptive learning rate and early stopping to ensure a better fit to the data.

We additionally compare with two different LLMs used in a single-model, zero-shot RAG architecture. In this setting, the LLMs are solely responsible for the glossing output, rather than correcting the output of a predecessor model.