Examining Exams Using Rasch Models and Assessment of Measurement Invariance

Statisticians often teach large lecture courses with introductions to statistics, probability, or mathematics in support of other curricula such as business and economics, social sciences, psychology, etc. Due to the large number of students and possibly also of lecturers who teach lectures and/or tutorials in parallel, it is often necessary to rely on exams and other assessments based on large pools of so-called closed (as opposed to open-ended) items, i.e., exercises which can be evaluated and graded automatically.

The most widely-used item type for such assessments are multiple-choice (also known as multiple-answer) or single-choice exercises. But due to the widespread adoption of learning management systems such as Moodle (Moodle), Canvas (Canvas), or Blackboard (Blackboard), especially since the Covid-19 pandemic, other item types are also being used increasingly. Often the evaluation is binary (correct vs. incorrect) but scores with partial credits for partially correct items are also frequently used.

Traditionally, mostly simple summary statistics have been used for the results from such large-scale exams, e.g., the proportion of students who correctly solved the different items and the number of items solved per student. However, recently there has also been increasing interest in so-called learning analytics which connects the results from different exams or assessments as well as covariates such as the field and duration of the study, prior knowledge from previous courses, etc., in order to better understand and shape the learning environments for the students (see Wiki+LearningAnalytics, for an overview and further references).

However, in Austria, to the best of our knowledge, it is still not common to apply standardized and/or automated psychometric assessments to exam results. A notable exception is the multiple-choice monitor at WU Wirtschaftsuniversität Wien introduced by Nettekoven+Ledermueller:2012; Nettekoven+Ledermueller:2014. In addition to various exploratory techniques they also employ probabilistic statistical models from psychometrics to gain more insights into exam results. More specifically, they use models from item response theory (IRT, Fischer+Molenaar:1995; VanDerLinden+Hambleton:1997), including the Rasch model (Rasch:1960) which is also employed in the analysis of international educational attainment studies such as PISA (Programme for International Student Assessment, https://www.oecd.org/pisa/).

Based on an exam’s item responses, IRT models can estimate various quantities of interest, most importantly the ability of the individual students and the difficulty of the different items (or exercises). A fundamental assumption is that the models’ parameters are invariant across all observations, which is also known as measurement invariance (see Horn+McArdle:1992, for an early overview in psychometrics). Otherwise observed differences in the items solved cannot be reliably attributed to the latent variable that the model purports to measure.

Typical sources for violation of measurement invariance in IRT models are multidimensionality (i.e., more than one latent variable instead of a single ability) or differential item functioning (see Debelak+Strobl:2024). The latter refers to the situation where the same item can be relatively easier or more difficult (compared to the remaining items) for different students, despite having the same latent ability.

In Section 2 we introduce a data set from one of our own mathematics courses, containing binary responses (correct vs. incorrect) from 13 items in an end-term exam of an introductory mathematics course for economics and business students. In Section LABEL:sec:irt the Rasch IRT model is briefly introduced, fitted to the data, and interpreted regarding the items’ difficulties and students’ abilities. Subsequently, in Section LABEL:sec:dif various methods for capturing violations of measurement invariance are applied: (1) Classical two-sample comparisons of two exogenously given groups along with modern methods for anchoring the item difficulty estimates. (2) Rasch trees based on generalized measurement invariance tests for data-driven detection of subgroups affected by DIF. (3) Rasch finite mixture models as an alternative way of data-driven characterization of DIF clusters. Section LABEL:sec:discussion wraps up the paper with a discussion and the epilogue Section LABEL:sec:epilogue concludes the paper by highlighting Fritz Leisch’s influence on different aspects of this work.

In all sections, emphasis is given to the hands-on application of the methods in \proglangR (R) – notably using the packages \pkgpsychotools (psychotools), \pkgpsychotree (psychotree), and \pkgpsychomix (psychomix) – along with the practical insights about the analyzed exam.