iolbench: Benchmarking LLMs on Linguistic Reasoning

The emergence of Large Language Models (LLMs) has led to growing interest in their capacity to perform reasoning tasks that extend beyond statistical pattern matching, particularly in domains requiring compositional generalization. Compositional reasoning involves deducing abstract rules and structures from a limited set of examples and applying these deductions to novel inputs—a process akin to human problem-solving. Linguistic problem-solving, especially in areas such as syntax, morphology, and phonology, offers a unique and rigorous benchmark for evaluating this capability. These tasks often involve complex linguistic phenomena, typologically diverse languages, and minimal data, requiring models to generalize effectively without relying on prior knowledge or memorized patterns.

In this work, we introduce iolbench, a novel dataset derived from the International Linguistics Olympiad (IOL), a competition designed to test advanced linguistic reasoning. Spanning over two decades of IOL problems, iolbench covers a wide range of linguistic phenomena, including phonological rule inference, morphological paradigm discovery, syntactic structure analysis, and semantic reasoning. These tasks are intentionally crafted to challenge models to hypothesize, test, and generalize abstract linguistic rules from limited data. By leveraging under-documented and typologically diverse languages (eg: Rotuman in Figure 1), iolbench minimizes reliance on pre-trained linguistic biases, providing a stringent evaluation of reasoning capabilities.

To evaluate state-of-the-art performance, we benchmark several leading LLMs, including OpenAI’s GPT-4 model family, Anthropic’s Claude models, and Google’s Gemini model, focusing on two key research questions: (1) To what extent can LLMs handle complex linguistic reasoning tasks requiring abstraction and generalization? (2) What specific strengths and weaknesses do these models exhibit in tackling linguistic problem-solving challenges? Our experiments analyze model performance across task types, identifying gaps in areas such as phonology and morphology, where systematic reasoning and hierarchical pattern recognition are critical. We further show performance on the text-only and multimodal splits of iolbench, demonstrating models are far less performant on visuo-linguistic problems.

By introducing iolbench and conducting a detailed evaluation of LLM performance, this work bridges the fields of computational linguistics and artificial intelligence, providing a foundation for developing models that better emulate the structured reasoning processes underlying human linguistic problem-solving.

There have been several recent efforts to evaluate the reasoning capabilities of LLMs through the development of domain-specific benchmarks, including MathBench (Liu et al., 2024) for mathematical problem-solving, SciBench (Wang et al., ) for scientific reasoning, and datasets based on competitive programming tasks that test logical and algorithmic thinking Veličković et al. (2022); Shi et al. (2024). These benchmarks provide valuable insights into the ability of LLMs to generalize abstract patterns and solve problems in structured domains. However, the domain of linguistic reasoning remains underexplored, despite its centrality to the applications and theoretical foundations of LLMs.

Language presents a uniquely challenging testbed for reasoning due to its compositional, hierarchical, and rule-governed nature. While models like GPT-4 have demonstrated proficiency in tasks requiring common-sense reasoning, textual coherence, and natural language generation, their capacity to navigate the intricacies of linguistic structure—particularly in typologically diverse or low-resource languages—remains insufficiently understood Dziri et al. (2024). This gap is particularly pronounced in areas such as morphology (e.g., rule-based inflectional systems) and syntax (e.g., hierarchical phrase structure), where solutions require reasoning over structured data rather than retrieving patterns from pre-trained distributions.

Existing benchmarks, such as BIG-Bench (Suzgun et al., 2023), include a handful of linguistics-inspired tasks but lack the depth and diversity required to evaluate models on truly complex linguistic reasoning. For instance, these tasks often focus on well-documented languages or simplified scenarios, which fail to capture the linguistic diversity and data sparsity challenges posed by real-world language problems. Other datasets, such as NLP-Bench (Song et al., ), focus on syntactic parsing or semantic role labeling but do not test a model’s ability to deduce and generalize linguistic rules from minimal examples—a hallmark of human linguistic reasoning.

The International Linguistics Olympiad (IOL) provides an ideal framework for addressing this gap. As one of the foremost competitions in linguistics, the IOL challenges participants to solve problems in a wide range of typologically diverse languages, often focusing on under-documented or low-resource languages. These problems require contestants to deduce complex grammatical rules, analyze morphophonemic patterns, and identify syntactic or semantic structures based on minimal data. The tasks are explicitly designed to test meta-linguistic reasoning and require abstract problem-solving skills rather than prior knowledge of the languages involved. As such, the IOL represents a uniquely challenging and unbiased benchmark for evaluating LLMs on tasks that align closely with human linguistic reasoning.

This work seeks to bridge the gap by introducing iolbench, a dataset derived from over 20 years of IOL problems. Unlike existing linguistics datasets, iolbench focuses on reasoning over typologically diverse languages, such as the endangered Aymara language. It also tests models’ ability to generalize rules across unseen data and to engage with tasks that combine multiple levels of linguistic analysis. By leveraging the rich, diverse, and challenging linguistic tasks provided by the IOL, we aim to address this gap and advance our understanding of LLMs’ ability to emulate the abstract reasoning processes characteristic of human linguistic cognition.

iolbench is a high-quality curated dataset of linguistic reasoning problems drawn from the International Linguistics Olympiad (IOL), a global competition that has convened annually since 2003. The IOL is distinguished by its focus on problem-solving abilities that are grounded in linguistic pattern recognition and hypothesis formation, rather than on encyclopedic knowledge of particular languages. Each problem requires participants to deduce underlying grammatical principles—pertaining to phonology, morphology, syntax, semantics, or orthography—from a minimal set of annotated examples. The reasoning process involves extracting abstract generalizations and applying inferred rules to novel test items.

The tasks are purposefully designed to be language-agnostic and frequently focus on low-resource languages where participants cannot rely on preexisting lexical or grammatical familiarity. Problems thus serve as a stringent evaluation framework for linguistic inference, and shallow statistical associations or memorized world knowledge are insufficient for successful performance. Instead, effective problem-solving requires modeling complex morphosyntactic systems, deducing phonological processes, and unraveling semantic and pragmatic relations with no pretraining bias toward these particular languages families.

Across this dataset, one observes a diverse typology of linguistic phenomena. For instance, phonological problems might involve identifying underlying phonemes, morphological tasks frequently center on discerning affixation patterns (e.g. verbal agreement paradigms), syntactic challenges may focus on constituent structure, semantic and lexicographical subtasks often demand recognizing compositional meanings within the provided language samples. All IOL problems are designed to allow inference of the solution from first principles, encouraging a reasoning-oriented approach to language analysis.

Our benchmarking of LLMs on iolbench, derived from the International Linguistics Olympiad, highlights both their strengths and limitations in linguistic reasoning. While GPT-4 outperformed other models overall, challenges persist in tasks involving morphology and phonology, where abstract rule induction and generalization are critical. These findings underscore the need for improved datasets that capture greater linguistic diversity and for advanced prompting strategies to enhance reasoning capabilities. Future work will focus on expanding iolbench and exploring tailored training approaches to address these gaps in linguistic problem-solving.

In future work, we plan to look at a larger number of models, including small language models and open language models. We also plan to explore fine-tuning of LLMs to improve their performance on iolbench. This work is meant to be a comprehensive novel exploration in benchmarking LLMs on problems from the International Linguistics Olympiad.