Causal Clustering for 1-Factor Measurement Models on Data with Various Types

For a binary variable $V_{i}$ generated as described above, its covariance with a Gaussian $X_{i}$ :

$Cov(V_{i},X_{i})=\int_{S_{i}}^{\infty}x\phi(x)dx=(\sqrt{2\pi})^{-1}exp({-S_{i}^{2}/2})$

where $\phi$ is the pdf of standard normal variable. For any other $X_{j}$ that forms a pure 1-factor measurement model, we have:

$Cov(V_{i},X_{j})=Cor(X_{i},X_{j})Cov(V_{i},X_{i})$

Since $X_{i}$ forms a pure triple with other measured Gaussian variables, replacing $X_{i}$ with $V_{i}$ does not change the pure property of those triples, i.e., adding another variable to the triple creates a vanishing tetrad regardless of the partition. Therefore, FOFC is able to identify them and correctly cluster $V_{i}$ with other measured Gaussian variables that form a pure 1-factor measurement model with $X_{i}$.

We can generalize this case: when $V_{i}$ is some function of $X_{i}$ and some independent noise $e$, i.e., $V_{i}=f(X_{i},e_{v})$ while other variables in the model are still linear Gaussian, the vanishing tetrad is still entailed by pure triple. If we denote the latent common cause as $L$ such that $X_{j}=a_{j}L+e_{j}$ for all $j$, we have:

Then the tetrad constraint holds for any partition:

$Cov(V_{i},X_{j})Cov(X_{h},X_{w})=Cov(V_{i},X_{h})Cov(X_{j},X_{w})=a_{j}a_{h}a_{w}\operatorname{\mathbb{E}}(Lf(X_{i},e_{v}))$

When several measured variables are binary while the latent variable is continuous, the measured variables would fail to form pure triples that entail the tetrad constraints. Assume two of the measured variables, $V_{i}$ and $V_{j}$, are binary and are generated by two standard Gaussian variables $X_{i}$ and $X_{j}$ as described before, and let $\Psi$ be the cdf of the joint standard bivariate normal distribution between $X_{i}$ and $X_{j}$ and $\rho_{ij}$ the correlation between $X_{i}$ and $X_{j}$, we get the covariance between $V_{i}$ and $V_{j}$:

Theoretically, FOFC is not entailed to build the right cluster, but the accuracy of FOFC clustering of measured variables with mixed types is too high to be chance. Similar to the last case, we can generalize this case: when $V_{i}$ is some function of $X_{i}$ and some noise $e$, i.e., $V_{i}=f(X_{i},e_{v_{i}})$ and $V_{j}$ is some function of $X_{j}$ and some noise $e$, i.e., $V_{j}=g(X_{j},e_{v_{j}})$ while other variables in the model are still linear Gaussian, one tetrad constraint is entailed. If we denote the latent common cause as $L$ such that $X_{h}=a_{h}L+e_{h}$ for all $h$, we have:

$Cov(V_{i},X_{h})=a_{h}\operatorname{\mathbb{E}}(Lf(X_{i},e_{v_{i}}))$

$Cov(V_{j},X_{h})=a_{h}\operatorname{\mathbb{E}}(Lg(X_{i},e_{v_{j}}))$

Then the tetrad constraint holds for the partition where the correlation between $V_{i}$ and $V_{j}$ is not included:

$Cov(V_{i},X_{w})Cov(V_{j},X_{h})=Cov(V_{i},X_{h})Cov(V_{j},X_{w})=a_{h}a_{w}\operatorname{\mathbb{E}}(Lf(X_{i},e_{v_{i}}))\operatorname{\mathbb{E}}((Lg(X_{i},e_{v_{j}}))$

Suppose a binary variable $V_{i}$ is generated from a median dichotomy, i.e., $S_{i}=0$. When every measured variables is generated from a median dichotomy, the covariance matrix of the measured data behaves quite similarly to a covariance matrix of the Gaussian variables. Given two standard Gaussian variables $u$ and $v$ with correlation $\rho$, one property of bivariate normal distribution formed by $u$ and $v$ is that:

$\dfrac{\partial\Psi(u,v,\rho)}{\partial\rho}=\psi(u,v,\rho)$

where $\psi(x,y,z)$ is the pdf of standard bivariate normal distributionMeyer (2013). Then the covariance between two measured binary variables is:

When $S_{i}=S_{j}=0$ the covariance can be written as :

The last step is based on the Taylor expansion of $arcsin$. Equation(3) indicates that the covariance between the two binary variables with median dichotomy is proportional to the correlation between the two mediate Gaussian variables. If $|\rho|\leq 0.65$, the difference between $arcsin(\rho)$ and $\rho$ is less than 10% Cox and Wermuth (2002). Since the covariance between the measured binary variables is approximately proportional to the correlation between the mediate Gaussian variables, the property that the a partition of any four mediate variables satisfies the tetrad constraints approximately holds on the measured binary variables. In fact, the performance of FOFC on binary data with median dichotomy is almost as good as on Gaussian data.

Recall that we assume the binary variable $V_{i}$ is generated from a standard Gaussian variable $X_{i}$ with a cutoff value $S_{i}$, such that $V_{i}=1$ if $X_{i}>S_{i}$ and $V_{i}=0$ otherwise. This section discusses the situation where $S_{i}\neq 0$. When the $|S_{i}|\leq 1$, FOFC is still reliable for building the right cluster. Recall the covariance between two binary variables:

When $r$ is small and the absolute value of $S_{i}$ and $S_{j}$ are smaller than 1 , $1-r^{2}$ is approximately 1 and $rS_{i}S_{j}$ is apporximately 0. The covariance can be approximated:

The final step of the approximation entails the binary variable with non-median dichotomy to satisfy the tetrad constraints.

Figure 3 shows how well binary variables derived from normal Gaussian with various dichotomies satisfy the tetrad constraints: in this figure, four binary variables whose original Gaussian variables form a pure 1-factor measurement model.are randomly partitioned twice; the y axis is the ratio of the product of correlation between these two partitions and x axis the largest absolute value the correlation the pairs of Gaussian variables that generates the binary variables in the partition can have. For the sake of convenience, all correlations here are positive since what matters is not being positive or negative but the absolute value. A ratio of 1 is when tetrad constraint holds. We can see that binary variables with median dichotomy satisfy the tetrad constraints with slight variations when the (absolute value of) the correlation between the standard Gaussian variables is less than 0.8; the product ratio of binary variables with non-median dichotomy fluctuates around “1” and become less and less stable as the correlation between the standard Gaussian variables increases, suggesting a susceptibility to external noises and complex connections between mediate Gaussian variables even if each of them forms a pure 1-factor measurement model..

The figure 3 is designed to plot Ratio versus the Largest Absolute Value of Continuous Correlation because, besides the absolute value of cutoff, (the absolute value of) continuous correlation is the only other thing that influences how well binary variables satisfy tetrad constraints. As mentioned in setion 3.3.1, the covariance between two binary variables is closer to proportional to the correlation of the two Gaussians that generate them when the Gaussian correlation is small. The closer to proportional the binary covariance is to the Gaussian correlation, the closer the binary tetrad containing the pair of binary variables satisfies the tetrad constraint (so the ratio is closer to 1) if other conditions are the same. In other words, after two partitions, the largest absolute value among the four continuous correlations determines how close the ratio is to ‘1’.

As shown above, the tetrad constraints approximately hold between discrete variables generated from continuous variables in a pure 1-factor measurement model.. The satisfaction of the tetrad constraints requires that correlations between the continuous variables are not too large and also the absolute values of cutoffs ($|S_{ic_{i}}|$) are small (to enable the approximation in 2.6 and 2.9).

On the other hand, the more categories a standard Gaussian variable is discretized into, the more similarly the resulting discrete variable behaves to the original Gaussian. Figure 4, which is a plot of the mean of ratio of the correlation between two discrete variables over the correlation between the two original Gaussian variables versus the number of categories a discrete variable takes, illustrates this feature.

The figure is generated by first calculating the mean of ratio: A 9x2 Matrix M is initialized to set the number of categories each pair of discrete variables are taking. $M$ represents 9 pairs of variables; one of the two variables in the ith pair takes $M[i,0]$ possible values and another $M[i,1]$ . The value of each entry of $M$ is set up like this: $M[i,1]=i+2$ $(i\in\{0...8\}$ ) and $M[i,0]$ is a random number between $2$ and $M[i,1]$. Let $i_{m}$ denote the variable that has more categories in $i$th pair and $\#(i_{m})$ the number of categories $i_{m}$ takes. The setup of $M$ entails that $\#(i_{m})>\#(j_{m})$ if $i>j$.

For a pair of discrete variables of which the numbers of possible values are $w$ and $z$, we calculate the correlation of the pair given the correlation between the Gaussian variables, as an element in ContCor, from which they are derived. Two cutoff arrays, one of length $w-1$ and another of $z-1$, were initialized such that each array has an increasing order and every two adjacent elements differ from a random number between 0 and 1. The two cutoff arrays are then centered to get a zero mean²²2the centering operation is just to make sure that some cutoffs are negative and some are positive; there is no difference in the final result depending on whether the cutoff array is centered or not. Given the correlation between the two standard Gaussian variables, the variance of each discrete variable is first calculated, then their covariance using the formula in remark, then their correlation, which is then divided by the Gaussian correlation to get the ratio. For every pair of discrete variables, such a ratio for every element in ContCor is calculated, then all the ratios are summed up and divided by the length of ContCor to get the mean of ratio. After collecting the mean of ratio for every pair of variable, the figure is plotted such that the x axis is the second column of $M$ and for a point with $M[j,1]$ as the x-coordinate, its y-coordinate is the mean of ratio for the ith pair.

As shown in this figure, the correlation between the discrete variables approaches the original Gaussian correlation closer and closer as the number of category increases. In fact, when the larger number of category exceeds six, the difference between the discrete correlation and continuous correlation is less than 10%.

This section presents the performance of FOFC on Gaussian data, binary data and data with more than two possible values discretized from the original continuous variables. The original continuous data is simulated from a graph with five latent variables each having four measured children. In the simulation study, the number of edges connecting the latents can be zero, one, three, six, seven or nine. Figure 6 shows one example where the graph has 5 pure 1-factor measurement models.with 6 edges connecting the five latents. These numbers are chosen to represent cases from those in which all latents are independent from each other to those in which the latents are densely connected. Note that the ability of the algorithm to detect impurities of the model, i.e., one measured variable is directly caused by more than one latent variables or measured variables directly causes other measured variables, is tested but is not explicitly shown, because such ability is found to be stable regardless of the data type. The graphs used for the simulation contains impurities, such as figure 5, where there are three pairs of measured variables having direct causal relations.

After the Gaussian data is simulated, if asked to discretize, the system will generate random cutoffs for each variable to linearly discretize the variable into the number of categories set up beforehand. The discrete data shown in this section belong to the following situations, each situation under three sample sizes(100, 500 and 2000):

• •

  binary variable from median dichotomy

• •

  binary variable from non-median dichotomy

• •

  data with THREE possible values

• •

  data with FOUR possible values

• •

  data with FIVE possible values

• •

  data with SIX possible values

• •

  data with EIGHT possible values

For the sake of convenience, the simulation is set up so that, when running on the discretized data, all variables included have the same number of possible values. This should not raise a concern because as proven in section 3.4, whether the measured discrete variables take the same number of possible values or not does not influence whether they satisfy the tetrad constraints.

The two indexes used to evaluate the performance of the algorithm are the very common ones: precision and recall. The performance for each index is shown by a figure.

One expects FOFC to identify latent variables through measured ones and ultimately recover the connection between the latents. Precision measures that, for each group of variables clustered by the algorithm, how accurate it is for them to actually form a pure 1-factor measurement model.: for an estimated cluster, the system will first find the true cluster that contains the largest number of the variables in this estimated cluster and calculate the percentage of variables in the estimated cluster contained in the true cluster as an individual precision for this single estimated cluster. In the best case where all the variables in an estimated cluster indeed forms the same pure 1-factor measurement model., the individual precision will be one; in the worst case the individual precision will be $1/n$ where $n$ is the number of variables in the estimated cluster. The system calculates the precision for each estimated cluster, then calculate the mean of all individual precisions as the final precision for the simulation.

Figure 7,9 and 11 shows the precision of FOFC performed on datasets with different sizes. The dataset with sample size 100 and 500 is simulated from graphs where there are 3, 6, or 9 edges connecting the five latent variables(such as figure 6). With sample size 2000, FOFC is tested on simulations where the true graph is in six different conditions. As mentioned before, the difference between the six conditions is the density of the connection between the five latent variables. The connection is introduced as the title of each of the six sub-figures. For each sub-figure, the x-axis is the type of data the algorithm runs on: 0 means the data is Gaussian; 2 indicates binary data with median dichotomy; $2\_$ binary data with non-median dichotomy; 3 data with three possible values; 4 data with four possible values, etc; the y-axis is the mean of final precision of the 40 simulations. Note that the 40 simulations are based on the same true graph.

The precision of FOFC increases as the sample size increases. The reliably high precision in the figure 11 indicates that, regardless of the situation of the data, if the algorithm detects a latent common cause from a group of variables by clustering them together, it is safe to say that these variables indeed share a latent common cause, forming a pure 1-factor measurement model..

Precision measures the accuracy of FOFC on different data types whereas recall measures the sensitivity, i.e., for each pure 1-factor measurement model.(or true cluster) in the true graph, how likely it is for the algorithm to actually put all the variables it contains into an estimated cluster: for an true cluster, the system will first find the estimated cluster that contains the largest number of the variables in this true cluster and calculate the percentage of variables in the true cluster contained in the estimated cluster as an individual recall for this single true cluster. In the best case where all the variables in an true cluster indeed are grouped into the same estimated cluster, the individual recall will be one; in the worst case the individual recall will be 0 when all variables in the true cluster are determined as independent from others. The system calculates the recall for each true cluster, then calculate the mean of all individual recalls as the final recall for the simulation.

Figure 8, 10 and 12 show the recall of the FOFC tested on the datasets from which the precisions with the corresponding sample sizes shown above are calculated. The connection between the five latent variables is introduced as the title of each of the six sub-figures. For each sub-figure, the x-axis has the same meaning as before; the y-axis is the mean of final recall from the 40 simulations each time.

The recall of the FOFC running on each data type is similar to precision except for the binary data with non-median dichotomy. As the number of edges connecting the five latent variables increases, the recall for non-median binary data decreases from 0.8 to less than 0.4, showing that the algorithm gradually loses the ability to identify pure clusters using the tetrad constraints. The high precision and low recall indicate that, when the data is non-median binary, FOFC can form correct clusters but can only recover some of them while falsely leaves many variables outside of clusters which they form in the true graph. A typical output of the algorithm in this situation is shown in figure 13 when the true graph is figure 6. Since the algorithm is run on binary data instead of the original continuous data, all the $X_{i}$ variables in figure 6 represents the Gaussian variable and in figure 13 the binary variable discretized from the original. $\_L_{i}$ in figure 13 represents the latent variable the algorithm postulated for each estimated cluster. Another thing worth mentioning is that, although FOFC is designed to study the latent variables, the algorithm itself does not calculate the connections between the latent variables and will not put edges between them.

As pointed in section 3.3.2, when the binary data is discretized with a random number between zero and one as the cutoff value whereas the original Gaussian variable has a mean of zero, although the correlation between the binary variables of which the parents form a pure 1-factor measurement model.approximately satisfy the tetrad constraints, the clustering decision is disturbed by the connections between the latent variables. The disturbance becomes inevitable for at least two reasons. First, the absolute value of the correlation shrinks after the dichotomy, making the algorithm more likely to judge two variable as independent even if they are not, not to mention testing the tetrad constraints. Second, the variables only approximately satisfy the tetrad constraints; as mentioned before, multiple factors can influence how well the constraints are satisfied, such as how large a cutoff value is. The variables not belonging to the same cluster will have non-zero correlation when the latent variables are not independent, making it harder for the algorithm to test whether certain group of variables satisfy the tetrad constraints or not.

MGM is an algorithm learning graphical structure over continuous and discrete variablesSedgewick et al. (2018). Unlike FOFC, which only relies on correlation, MGM uses a pairwise graphical modelLee and Hastie (2015):

$p(x,y;\Theta)\propto exp(\sum\limits_{s=1}^{p}\sum\limits_{t=1}^{p}-\dfrac{1}{2}\beta_{st}x_{s}x_{t}+\sum\limits_{s=1}^{p}\alpha_{s}x_{s}+\sum\limits_{s=1}^{p}\sum\limits_{j=1}^{q}\rho_{sj}(y_{j})x_{s}+\sum\limits_{j=1}^{q}\sum\limits_{r=1}^{q}\theta_{rj}(y_{r},y_{j}))$

where there $p$ continuous variables $x$ and $q$ discrete varaibles $y$.

Using the pseudolikelihood method, MGM estimates the parameters $\Theta=[\{\beta_{st}\},\{\alpha_{s}\},\{\rho_{sj}\},\{\theta_{rj}\}]$ and use them to build an undirected graph:

• •

  an edge is added between two nodes $s$ and $t$ representing continuous variables $x_{s}$ and $x_{t}$ if $\beta_{st}\neq 0$

• •

  an edge is added between two nodes $s$ and $j$ representing continuous variable $x_{s}$ and discrete variable $y_{j}$ if it’s not the case where $\rho_{sj}=0$ for all values of $y_{j}$

• •

  an edge is added between two nodes $r$ and $j$ representing discrete variables $y_{r}$ and $y_{j}$ if $\theta_{rj}\neq 0$ for all values of $y_{j}$ and $y_{r}$

The undirected graph is built with the idea that the absence of an edge between two nodes $x$ and $y$ means that variables $x$ and $y$ are independent conditioning on all other variables. This undirected graph is then pruned using an independence test. For instance, when running on a dataset with three variables where the true graph is figure 18, the MGM will first generate figure 19 since any two variables are dependent given the third one. The final output will be a undirected graph with the edge between $X_{2}$ and $X_{3}$ removed since they are marginally independent.

The recall and precision used to evaluate the FOFC are the same as before, but customized for MGM.

MGM does not decide whether some variables share a latent common cause, but we can interpret its output in that way: if some variables form a clique, this clique is equivalent to an estimated cluster from FOFC. Therefore, the recall can be calculated by checking whether all variables from a true cluster form a clique in the MGM output. Similar to the calculation for FOFC, individual recall is calculated for each true cluster, which will be either 1 if a clique is formed or 0 if not. After this step, final recall is calculated as the mean of individual recall.

The precision for MGM is expected to measure that, for each group of variables forming a clique, how accurate it is for them to actually form a pure 1-factor measurement model.in the true graph. In order to do that, the system will first check if the variables forming a pure 1-factor measurement model.in the true graph forms a clique in the MGM output, then will build groups of variables called fake clusters. Each fake cluster has the same size as each true cluster, but variables in it do not belong to the same pure 1-factor measurement model.in the true graph. For instance, given the true graph figure 20, the system will not only check whether variables in a true cluster, such as $\{X_{1},X_{2},X_{3},X_{4}\}$ form a clique in the MGM output, it will also check whether variables in a fake cluster, such as $\{X_{1},X_{5},X_{12},X_{2}\}$, form a clique.

The precision is calculated based on the formula of precision:

$precision=\dfrac{tp}{tp+fp}$

where $tp$ stands for “true positive” and $fp$ “false positive”. For each MGM output, the number of true clusters that are cliques in the output is treated as the $tp$ and the number of fake clusters that are cliques in the output is treated as the $fp$. Therefore, for each MGM output, in the best case the precision will be 1 and worse case 0. If the MGM output is a complete graph, the precision is 0.5 since all true and fake clusters are all cliques, i.e., $tp=fp$.

Figure 20 shows the graph used for simulation. After the Gaussian data is generated, it will first be discretized with random cutoff value, then mixed with the original Gaussian data such that in each pure cluster, half of the measured variables are continuous and the other half are discrete. Figure 21 shows the result of the comparison. For each sub-figure the x-axis is the number of possible values the discrete variable in the mixed dataset can take: to distinguish the two algorithms, $i$ means the discrete variable can take $i$ possible values and the mixed dataset is run by FOFC; $i^{\prime}$ means $i$ possible values and is run by MGM; the y-axis is the mean of precision (recall) of 40 simulations with 2000 data points each time. Note that the 40 simulations are based on the same true graph.

To make the comparison explicit, the performances of each algorithm on the same mixed dataset are put next to each other. The blue bar refers to FOFC and the orange MGM.

Figure 21 shows that the recall of FOFC and MGM are high, but MGM is better than FOFC. It can be caused by the fact that FOFC is susceptible to the shrinking of correlation due to discretization. The difference between the precision of FOFC and MGM is rather obvious.

The precision of MGM being between 0.6 and 0.7 indicates that, when the latent variables are not independent, MGM has a hard time deciding whether variables are from the same pure cluster or not. In fact, the precision is consistently 0.5 when the true graph has two pure 1-factor measurement models.and the one latent variable is a direct cause of another. If the latent variables are independent, the MGM performs as well as, or slightly better in terms of recall than the FOFC.