Latent class analysis with weighted responses

Latent class model (LCM) [1, 2, 3] is a powerful tool for categorical data, with many applications across various areas such as social, psychological, behavioral, and biological sciences. These applications include movie rating [4, 5], psychiatric evaluation [6, 7, 8, 9], educational assessments [10], political surveys [11, 12, 13, 14], transport economics personal interview [15], and disease etiology detection [16, 17, 18]. In categorical data, subjects (individuals) typically respond to several items (questions). LCM is a theoretical model that categorizes subjects into disjoint groups, known as latent classes, according to their response pattern to a collection of categorical items. For example, in movie rating, latent classes may represent different groups of users with an affinity for certain movie themes; in psychological tests, latent classes may represent different types of personalities. In educational assessments, latent classes may indicate different levels of abilities. In political surveys, latent classes may represent distinct types of political ideologies. In transport economics personal interview, each latent class stands for a partition of the population. In disease etiology detection, latent classes may represent different disease categories. To infer latent classes for categorical data generated from LCM, various approaches have been developed in recent years, including maximum likelihood estimation techniques [19, 20, 21, 22] and tensor-based methods [23, 24].

To mathematically describe categorical data, let $R$ be the $N$-by-$J$ observed response matrix such that $R(i,j)$ represents subject $i$’s response to item $j$, where $N$ denotes the number of subjects and $J$ denotes the number of items. For LCM, researchers commonly focus on binary choice data where elements of the observed response matrix $R$ only take 0 or 1 [16, 10, 24, 25, 26, 27, 28, 29, 30, 31, 32]. LCM models binary response matrix by generating its elements from a Bernoulli distribution. In categorical data, binary responses can be agree/disagree responses in psychiatric evaluation, correct/wrong responses in educational assessments, and presence/absence of symptoms in disease etiology detection. However, categorical data is more than binary response. Categorical data with weighted responses is also commonly encountered in the real world and ignoring weighted data may lose potentially meaningful information [33]. For example, in movie rating [4], rating scores range in $\{1,2,3,4,5\}$ and simply letting $R$ be binary by recording rated/not rated loses valuable information that can reflect users’ preference patterns; for real-world categorical data from various online personality tests in the link https://openpsychometrics.org/_rawdata/, the range of most responses are $\{0,1,2,\ldots,m\}$, where $m$ is an integer like 2, 5, and 10; in the buyer-seller rating e-commerce data [34], elements of the observed response matrix take values in $\{-1,0,1\}$ (for convenience, we call such $R$ as signed response matrix in this paper) since sellers are rated by users by applying three levels of rating, “Positive”, “Neutral”, and “Negative”. In the users-jokes ratting categorical data Jester 100 [35], all responses (i.e., ratings) are continuous numbers ranging in $[-10,10]$. All aforementioned real-world data with weighted responses cannot be generated from a Bernoulli distribution. Therefore, the classical latent class model is inadequate for handling the aforementioned data with weighted responses. As a result, it is desirable to develop a more flexible model for data with weighted responses. With this motivation, our key contributions to the literature of latent class analysis are summarized as follows.

• 1.

  Model. We propose a novel, identifiable, and generative statistical model, the weighted latent class model (WLCM), for categorical data with weighted responses, where the responses can be continuous or negative values. Our WLCM allows the elements of an observed weighted response matrix $R$ to be generated from any distribution provided that the population version of $R$ under WLCM enjoys a latent class structure. For example, our WLCM allows $R$ to be generated from Bernoulli, Normal, Poisson, Binomial, Uniform, and Exponential distributions, etc. By considering a specifically designed discrete distribution, our WLCM can also model signed response matrices. For details, please refer to Examples 1-7. For comparison, LCM requires $R$ to be generated from Bernoulli distribution and LCM is a sub-model of our WLCM. Under the proposed model, the elements of the observed weighted response matrix $R$ can take any value. Therefore, WLCM is more flexible than LCM. As far as we know, our WLCM is the first statistical model for categorical data in which weighted responses can be continuous or negative values.

• 2.

  Algorithm. We develop an easy-to-implement algorithm, spectral clustering with K-means (SCK), to infer latent classes for weighted response matrices generated from arbitrary distribution under the proposed model. Our algorithm is designed based on a combination of two popular techniques: the singular value decomposition (SVD) and the K-means algorithm.

• 3.

  Theoretical property. We build a theoretical framework to show that SCK enjoys consistent estimation under WLCM. We also provide Examples 1-7 to show that the theoretical performance of the proposed algorithm can be different when the observed weighted response matrices $R$ are generated from different distributions under the proposed model.

• 4.

  Empirical validation. We conduct extensive simulations to validate our theoretical insights. Additionally, we apply our SCK approach to two real-world datasets with meaningful interpretations.

The remainder of this paper is organized as follows. Section 2 describes the model. Section 3 details the algorithm. Section 4 establishes the consistency results and provides examples for further analysis. Section 5 contains numerical studies that verify our theoretical findings and examine the performance of the proposed method. Section 6 demonstrates the proposed method using two real-world datasets. Section 7 concludes the paper with a brief discussion of contributions and future work.

The following notations will be used throughout the paper. For any positive integer $m$, let $[m]$ and $I_{m\times m}$ be $[m]:=\{1,2,\ldots,m\}$ and the $m\times m$ identity matrix, respectively. For any vector $x$ and any $q>0$, $\|x\|_{q}$ denotes $x$’s $l_{q}$-norm. For any matrix $M$, $M^{\prime}$ denotes its transpose, $\|M\|$ denotes its spectral norm, $\|M\|_{F}$ denotes its Frobenius norm, $\mathrm{rank}(M)$ denotes its rank, $\sigma_{i}(M)$ denotes its $i$-th largest singular value, $\lambda_{i}(M)$ denotes its $i$-th largest eigenvalue ordered by magnitude, $M(i,:)$ denotes its $i$-th row, and $M(:,j)$ denotes its $j$-th column. Let $\mathbb{R}$ and $\mathbb{N}$ be the set of real numbers and nonnegative integers, respectively. For any random variable $X$, $\mathbb{E}(X)$ and $\mathbb{P}(X=a)$ are the expectation and the probability that $X$ equals to $a$, respectively. Let $\mathbb{M}_{m,K}$ be the collection of all $m\times K$ matrices where each row has only one $1$ and all others $0$.

Unlike most researchers that only focus on binary responses, in our weighted response setting in this paper, all elements of the observed weighted response matrix $R$ are allowed to be any real value, i.e., $R\in\mathbb{R}^{N\times J}$.

Consider categorical data with $N$ subjects and $J$ items, where the $N$ subjects belong to $K$ disjoint extreme latent profiles (also known as latent classes). Throughout this paper, the number of classes $K$ is assumed to be a known integer. To describe the membership of each subject, we let $Z$ be a $N\times K$ matrix such that $Z(i,k)$ is 1 if subject $i$ belongs to the $k$-th extreme latent profile and $Z(i,k)$ is 0 otherwise. Call $Z$ the classification matrix in this paper. For each subject $i\in[N]$, it is assumed to belong to a single extreme latent profile. For convenience, define $\ell$ as a $N$-by-$1$ vector whose $i$-th entry $\ell(i)$ is $k$ if the $i$-th subject belongs to the $k$-th extreme latent profile for $i\in[N]$. Thus for subject $i\in[N]$, we have $Z(i,\ell(i))=1$ and the other $(K-1)$ entries of the $K\times 1$ classification vector $Z(i,:)$ is 0.

Introduce the $J\times K$ item parameter matrix $\Theta\in\mathbb{R}^{J\times K}$. For $k\in[K]$, our weighted latent class model (WLCM) assumes that $\Theta(j,k)$ collects the conditional-response expectation for the response of the $i$-th subject to the $j$-th item under arbitrary distribution $\mathcal{F}$ provided that subject $i$ belongs to the $k$-th extreme latent profile. Specifically, for $i\in[N],j\in[J]$, given the classification vector $Z(i,:)$ of subject $i$ and the item parameter matrix $\Theta$, our WLCM assumes that for arbitrary distribution $\mathcal{F}$, the conditional response expectation of the $i$-th subject to the $j$-th item is

Based on Equation (1), our WLCM can be simplified as follows.

Definition 1 says that WLCM is determined by the classification matrix $Z$, the item parameter matrix $\Theta$, and the distribution $\mathcal{F}$. For brevity, we denote WLCM by $WLCM(Z,\Theta,\mathcal{F})$. Under WLCM, $\mathcal{F}$ is allowed to be any distribution as long as Equation (2) is satisfied under $\mathcal{F}$, i.e., WLCM only requires the expectation (i.e., population) response matrix $R_{0}$ of the observed weighted response matrix $R$ to be $Z\Theta^{\prime}$ under any distribution $\mathcal{F}$.

The following proposition shows that the WLCM model is identifiable as long as there exists at least one subject for every extreme latent profile.

All proofs of theoretical results developed in this paper are given in the Appendix. The condition that each extreme latent profile must contain at least one subject means that each extreme latent profile cannot be an empty set and we have $\mathrm{rank}(Z)=K$.

The observed weighted response matrix $R$ along with the ground-truth classification matrix $Z$ and the item parameter matrix $\Theta$ can be generated using our WLCM model as follows: let $R(i,j)$ be a random variable generated by distribution $\mathcal{F}$ with expected value $R_{0}(i,j)$ for $i\in[N],j\in[J]$, where $R_{0}=Z\Theta^{\prime}$ satisfies the latent structure required by WLCM. In latent class analysis, given the observed weighted response matrix $R$ generated from $WLCM(Z,\Theta,\mathcal{F})$, our goal is to infer the classification matrix $Z$ and the item parameter matrix $\Theta$. Proposition 1 ensures that the model parameters $Z$ and $\Theta$ can be reliably inferred from the observed weighted response matrix $R$. In the following two sections, we will develop a spectral algorithm to fit WLCM and show that this algorithm yields consistent estimation.

We have presented our model, WLCM, and demonstrated its superiority over the classical latent class model. In addition to providing a more general model for latent class analysis, we are also interested in estimating the model parameters. In this section, we focus on the parameter estimation problem within the WLCM framework by developing an efficient and easy-to-implement spectral method.

To provide insight into developing an algorithm for the WLCM model, we first consider an oracle case where we observe the expectation response matrix $R_{0}$ given in Equation (2). We would like to estimate $Z$ and $\Theta$ from $R_{0}$. Recall that the item parameter matrix $\Theta$ is a $J$-by-$K$ matrix , here we let $\mathrm{rank}(\Theta)=K_{0}$, where $K_{0}$ is a positive integer and it is no larger than $K$. As $R_{0}=Z\Theta^{\prime},\mathrm{rank}(Z)=K$, and $\mathrm{rank}(\Theta)=K_{0}\leq K$, we see that $R_{0}$ is a rank-$K_{0}$ matrix. As the number of extreme latent profiles $K$ is usually far smaller than the number of subjects $N$ and the number of items $J$, the $N$-by-$J$ population response matrix $R_{0}$ enjoys a low-dimensional structure. Next, we will demonstrate that we can greatly benefit from the low-dimensional structure of $R_{0}$ when we aim to develop a method to infer model parameters under the WLCM model.

Let $R_{0}=U\Sigma V^{\prime}$ be the compact singular value decomposition (SVD) of $R_{0}$ such that $\Sigma$ is a $K_{0}\times K_{0}$ diagonal matrix collecting the $K_{0}$ nonzero singular values of $R_{0}$. Write $\Sigma=\mathrm{diag}(\sigma_{1}(R_{0}),\sigma_{2}(R_{0}),\ldots,\sigma_{K_{0}}(R_{0}))$. The $N\times K_{0}$ matrix $U$ collects the corresponding left singular vectors and it satisfies $U^{\prime}U=I_{K_{0}\times K_{0}}$. Similarly, the $J\times K_{0}$ matrix $V$ collects the corresponding right singular vectors and it satisfies $V^{\prime}V=I_{K_{0}\times K_{0}}$. For $k\in[K]$, let $N_{k}$ be the number of subjects that belong to the $k$-th extreme latent profile, i.e., $N_{k}=\sum_{i=1}^{N}Z(i,k)$. The ensuing lemma constitutes the foundation of our estimation method.

From now on, for the simplicity of our further analysis, we let $K_{0}\equiv K$. Hence, the last statement of Lemma 1 always holds.

The second statement of Lemma 1 indicates that the rows of $U$ corresponding to subjects assigned to the same extreme latent profile are identical. This circumstance implies that the application of a clustering algorithm to the rows of $U$ can yield an exact reconstruction of the classification matrix $Z$ after a permutation of the $K$ extreme latent profiles.

In this paper, we adopt the K-means clustering algorithm, an unsupervised learning technique that groups similar data points into $K$ clusters. This clustering technique is detailed as follows,

where $\bar{U}$ is any $N\times K$ matrix. For convenience, call Equation (6) as “Run K-means algorithm on all rows of $\bar{U}$ with $K$ clusters to obtain $\bar{\bar{Z}}$” because we are interested in the classification matrix $\bar{\bar{Z}}$. Let $\bar{U}$ in Equation (6) be $U$, the second statement of Lemma 1 guarantees that $\bar{\bar{Z}}=Z\mathcal{P},\bar{\bar{X}}=\mathcal{P}^{\prime}X$, where $\mathcal{P}$ is a $K\times K$ permutation matrix, i.e., running K-means algorithm on all rows of $U$ exactly recovers $Z$ up to a permutation of the $K$ extreme latent profiles.

After obtaining $Z$ from $U$, $\Theta$ can be recovered subsequently by Equation (4). The above analysis suggests the following algorithm, Ideal SCK, where SCK stands for Spectral Clustering with K-means. Ideal SCK returns a permutation of $(Z,\Theta)$, which also supports the identifiability of the proposed model as stated in Proposition 1.

For the real case, the weighted response matrix $R$ is observed rather than the expectation response matrix $R_{0}$. We now move from the ideal scenario to the real scenario, intending to estimate $Z$ and $\Theta$ when the observed weighted response matrix $R$ is a random matrix generated from an unknown distribution $\mathcal{F}$ satisfying Equation (2) with $K$ extreme latent profiles under the WLCM model. The expectation of $R$ is $R_{0}$ according to Equation (2) under WLCM, so intuitively, the singular values and singular vectors of $R$ will be close to those of $R_{0}$. Set $\hat{R}=\hat{U}\hat{\Sigma}\hat{V}^{\prime}$ as the top $K$ SVD of $R$, where $\hat{\Sigma}$ is a $K\times K$ diagonal matrix collecting the top $K$ singular values of $R$. Write $\hat{\Sigma}=\mathrm{diag}(\sigma_{1}(R),\sigma_{2}(R),\ldots,\sigma_{K}(R))$. As $\mathbb{E}(R)=R_{0}$ and the $N\times J$ matrix $R_{0}$ has $K$ non-zero singular values while the other $(\mathrm{min}(N,J)-K)$ singular values are zeros, we see that $\hat{R}$ should be a good approximation of $R_{0}$. Matrices $\hat{U}\in\mathbb{R}^{N\times K},\hat{V}\in\mathbb{R}^{J\times K}$ collect the corresponding left and right singular vectors and satisfy $\hat{U}^{\prime}\hat{U}=\hat{V}^{\prime}\hat{V}=I_{K\times K}$. The above analysis implies that $\hat{U}$ should have roughly $K$ distinct rows because $\hat{U}$ is a slightly perturbed version of $U$. Therefore, to obtain a good estimation of the classification matrix $Z$, we should apply the K-means algorithm on all rows of $\hat{U}$ with $K$ clusters. Let $\hat{Z}$ be the estimated classification matrix returned by applying the K-means method on all rows of $\hat{U}$ with $K$ clusters. Then we are able to obtain a good estimation of $\Theta$ according to Equation (4) by setting $\hat{\Theta}=\hat{V}\hat{\Sigma}\hat{U}^{\prime}\hat{Z}(\hat{Z}^{\prime}\hat{Z})^{-1}$. Algorithm 2, referred to as SCK, is a natural extension of the Ideal SCK from the oracle case to the real case. Note that in our SCK algorithm, there are only two inputs: the observed weighted response matrix $R$ and the number of latent classes $K$, i.e., SCK does not require any tuning parameters.

Here, we evaluate the computational cost of our SCK algorithm. The computational cost of the SVD step involved in the SCK approach is $O(\max(N^{2},J^{2})K)$. For the K-means algorithm, its complexity is $O(NlK^{2})$ with $l$ being the number of K-means iterations. In all experimental studies considered in this paper, $l$ is set as 100 for the K-means algorithm. The complexity of the last step in SCK is $O(JNK)$. Since $K\ll\min(N,J)$ in this paper, as a consequence, the total time complexity of our SCK algorithm is $O(\max(N^{2},J^{2})K)$.

In this section, we present comprehensive theoretical properties of the SCK algorithm when the observed weighted response matrix $R$ is generated from the proposed model. Our objective is to demonstrate that the estimated classification matrix $\hat{Z}$ and the estimated item parameter matrix $\hat{\Theta}$ both concentrate around the true classification matrix $Z$ and the true item parameter matrix $\Theta$, respectively.

Let $\mathcal{T}=\{\mathcal{T}_{1},\mathcal{T}_{2},\ldots,\mathcal{T}_{K}\}$ be the collection of true partitions for all subjects, where $\mathcal{T}_{k}=\{i:Z(i,k)=1\mathrm{~{}for~{}}i\in[N]\}$ for $k\in[K]$, i.e., $\mathcal{T}_{k}$ is the set of true partition of subjects into the $k$-th extreme latent profile. Similarly, let $\hat{\mathcal{T}}=\{\hat{\mathcal{T}}_{1},\hat{\mathcal{T}}_{2},\ldots,\hat{\mathcal{T}}_{K}\}$ represent the collection of estimated partitions for all subjects, where $\hat{\mathcal{T}}_{k}=\{i:\hat{Z}(i,k)=1\mathrm{~{}for~{}}i\in[N]\}$ for $k\in[K]$. We use the measure defined in [36] to quantify the closeness of the estimated partition $\hat{\mathcal{T}}$ and the ground truth partition $\mathcal{T}$. Denote the Clustering error associated with $\mathcal{T}$ and $\hat{\mathcal{T}}$ as

where $S_{K}$ represents the set of all permutations of $\{1,2,\ldots,K\}$, $\mathcal{\hat{\mathcal{T}}}^{c}_{\pi(k)}$ and $\mathcal{T}^{c}_{k}$ denote the complementary sets. As stated in the reference [36], $\hat{f}$ evaluates the maximum proportion of subjects in the symmetric difference of $\mathcal{T}_{k}$ and $\hat{\mathcal{T}}_{\pi(k)}$. Since the observed weighted response matrix $R$ is generated from WLCM with expectation $R_{0}$, and $\hat{f}$ measures the performance of the SCK algorithm, it is expected that SCK estimates $Z$ with small Clustering error $\hat{f}$.

For convenience, let $\rho=\mathrm{max}_{j\in[J],k\in[K]}|\Theta(j,k)|$ and call it the scaling parameter. Let $B=\frac{\Theta}{\rho}$, we have $\mathrm{max}_{j\in[J],k\in[K]}|B(j,k)|=1$ and $R_{0}=\rho ZB^{\prime}$. Let $\tau=\mathrm{max}_{i\in[N],j\in[J]}|R(i,j)-R_{0}(i,j)|$ and $\gamma=\mathrm{max}_{i\in[N],j\in[J]}\mathrm{Var}(R(i,j))$ where $\mathrm{Var}(R(i,j))$ means the variance of $R(i,j)$. We require the following assumption to establish theoretical guarantees of consistency for our SCK method.

The following theorem presents our main result, which provides upper bounds for the error rates of our SCK algorithm under our WLCM model.

Because our WLCM is distribution-free, Theorem 1 provides a general theoretical guarantee of the SCK algorithm when $R$ is generated from WLCM for any distribution $\mathcal{F}$ as long as Equation (2) is satisfied. We can simplify Theorem 1 by considering additional conditions:

For the case $J=\beta N$ for any positive constant $\beta$, Corollary 1 implies that the SCK algorithm yields consistent estimation under WLCM since the error bounds in Corollary 1 decrease to zero as $N\rightarrow+\infty$ when $\rho$ and distribution $\mathcal{F}$ are fixed.

Recall that $R$ is an observed weighted response matrix generated from a distribution $\mathcal{F}$ with expectation $R_{0}=Z\Theta^{\prime}=\rho ZB^{\prime}$ under the WLCM model and $\gamma$ is the maximum variance of $R(i,j)$ and it is closely related to the distribution $\mathcal{F}$, the ranges of $R,\rho,B$, and $\gamma$ can vary depending on the specific distribution $\mathcal{F}$. The following examples provide the ranges of $R,\rho,B$, the upper bound of $\gamma$, and the explicit forms of error bounds in Theorem 1 for different distribution $\mathcal{F}$ under our WLCM model. Meanwhile, based on the explicitly derived error bounds for different distribution $\mathcal{F}$, we also investigate how the scaling parameter $\rho$ influences the performance of the SCK algorithm in these examples. For all pairs $(i,j)$ with $i\in[N],j\in[J]$, we consider the following distributions when $\mathbb{E}(R)=R_{0}$ in Equation (2) holds.