Assessment of Misspecification in CDMs using a Generalized Information Matrix Test

In the statistical machine learning framework, the Data Generating Process (DGP) generates observable data directly from an underlying unobservable probability distribution, $p_{\text{DGP}}$. A probability model,$M$, is a collection of probability mass functions. If $p_{DGP}\in M$, then $M$ is correctly specified; otherwise, $M$ is misspecified with respect to $p_{\text{DGP}}$. Less formally, a model capable of representing the DGP is called a ”correctly specified model.”

The parameter prior for the two-dimensional parameter vector $\boldsymbol{\beta}_{j}$, associated with the $j$th question, is represented by a bivariate Gaussian density, denoted as $p(\boldsymbol{\beta}_{j})$. This density has a two-dimensional mean vector $\boldsymbol{\mu}_{j}$ and a two-dimensional covariance matrix $\bf C_{\boldsymbol{\beta}}$ for $j=1,\ldots,J$. It is assumed that the constants $\mu_{1},\ldots,\mu_{J}$ are known, and $\boldsymbol{C}_{\boldsymbol{\beta}}$ is a positive number. Let $\boldsymbol{\mu}_{\beta}=[\mu_{1},\ldots,\mu_{J}]$. The joint distribution of $\boldsymbol{\beta}=[\boldsymbol{\beta}_{1},\ldots,\boldsymbol{\beta}_{J}]$ is specified as the parameter prior:

The likelihood function of the response vector of examinee $i$ who is assumed to have attribute pattern ${\boldsymbol{\alpha}}$ is given by:

yielding the MAP empirical risk function:

where

In this study, the MAP empirical risk function (3) involves summing over all possible latent skill attribute patterns. Assume $\hat{\ell}_{n}$ is a twice-continuously differentiable objective function. Once a critical point is reached, then the Hessian of $\hat{\ell}_{n}$ can be evaluated at that point to check if the critical point is a strict local minimizer (e.g., Golden2020 (20)). Here we assume that a parameter estimate $\hat{\boldsymbol{\beta}}_{n}$ has been obtained which is a strict local minimizer of $\hat{\ell}_{n}({\boldsymbol{\beta}})$ in some (possibly very small) closed, bounded, and convex region of the parameter space $\Omega$ for all sufficiently large $n$. Furthermore, we assume that the expected value of $\hat{\ell}_{n}({\boldsymbol{\beta}})$, $\ell({\boldsymbol{\beta}})$, has a strict global minimizer, ${\boldsymbol{\beta}}^{*}$, in the interior of $\Omega$. This setup of the problem thus allows for situations where $\ell$ has multiple minimizers, maximizers, and saddlepoints over the entire unrestricted parameter space. Given these assumptions, it can be shown (e.g., Wh82 (30, 20)) that $\hat{\boldsymbol{\beta}}_{n}$ in this case is a consistent estimator of ${\boldsymbol{\beta}}^{*}$. In addition, for large sample sizes, the effects of the parameter prior becomes negligible and the MAP estimate has the same asymptotic distribution as the maximum likelihood estimate.

The simulation studies used CDMs with parameters estimated from the Ta84 (29) fraction-subtraction data set. The dataset consists of a set of math problems involving fractions subtraction, designed to assess students’ ability to solve problems involving fractions. In this case, we utilized the ’Fraction.1’ dataset from the ’CDM’ R package (cdmR (19)) containing the dichotomous responses of 536 middle school students over 15 fraction subtraction test items. The $\bf Q$-matrix specifying which of five distinct skills were required to answer a particular test item was based upon the $\bf Q$-matrix used by Ta84 (29) and provided in cdmR (19).

First, the described DINA CDM was fit to the data set with a sample size of $n=536$ using the previously described MAP estimation method (see Equation 3). A multivariate Gaussian prior with mean vector $\boldsymbol{\mu}_{\boldsymbol{\beta}}$ and covariance matrix $\bf C_{\boldsymbol{\beta}}$ was used for all items (see Equation 1). To ensure guess (g) and slip (s) probabilities both equaled 0.354 for a specific item, the Gaussian Parameter Prior mean vector for that item was set as $\boldsymbol{\mu}_{\boldsymbol{\beta}}=\begin{bmatrix}1.2&0.6\end{bmatrix}^{T}$. An uninformative Gaussian Parameter Prior covariance matrix $\bf C_{\boldsymbol{\beta}}$ was chosen to be $\sigma^{2}{\bf I}_{2}$, where $\sigma^{2}=4500$. The fitted DINA CDM will be considered as the Data Generating Process CDM ($DGP_{CDM}$). To create a scenario in which the CDM is correctly specified, we refitted the initial original DINA CDM by sampling from the original fitted DINA CDM. Five different versions of the original DINA CDM, corresponding to ”misspecified models,” were used. These versions were created by flipping Tatsuoka’s $\bf Q$-matrix elements with different probabilities (0%, 1%, 5%, 10%, 15%, and 20%). The 0% misspecification level corresponds to the correctly specified case. For each level of $\bf Q$-matrix misspecification, we generate 5 different misspecified Q matrices at that level, allowing for variations in which elements are altered. We refer to a specific way of misspecifying the $\bf Q$-matrix at a given level as a ”replication.” Then, we fit each of the 5 replications of the $\bf Q$-matrix at a particular misspecification level to 50 bootstrap-generated datasets from the original DINA CDM.

We computed $GIMT_{Det}$ statistics as a function of misspecification level. Then we evaluated the performance of the $GIMT_{Det}$ in classifying correctly specified and misspecified models using the Receiver Operating Characteristic (ROC) curve. The ROC curve plots the true positive rate against the false positive rate at different decision thresholds, allowing for an investigation of the discrimination performance of the $GIMT_{Det}$. Additionally, the area under the ROC curve (AUROC) can provide a quantitative measure of the $GIMT_{Det}$ discrimination, with a high AUROC value indicating that the $GIMT_{Det}$ effectively distinguishes between correctly specified and misspecified models. Conversely, an AUROC close to 0.5 indicates no discrimination performance.

Table 1 displays the AUROC values under various misspecification levels which shows that AUROC values increase as the misspecification level increases. The results of the simulation study are also presented in Figure 1, which displays how the ROC curves evolve at various misspecification levels (0%, 1%, 5%, 10%, 15%, and 20%) with a constant sample size of n=536. At the 20% misspecification level, the ROC curve demonstrates good discrimination performance, closely approaching the upper-left corner.

When the misspecification level was greater than 5%, the GIMT showed good discrimination performance regardless of the manner in which the Q matrix was decimated. In addition, the GIMT did not detect model misspecification for the correctly specified case (0%) and did not detect model misspecification for the slightly misspecified case (1%). For moderate misspecification levels (5%), however, only two of the five replications showed effective discrimination performance. There was also substantial variability across replications for the 5% case. Currently, we view the 5% level as a ”transition region” and we plan to investigate this further using additional replications. In summary, the new GIMT method for misspecification detection in CDMs appears to be promising. Further research will continue to explore the capabilities and limitations of this approach.