Title

We consider a generative model for a dataset $Y$ with $N$ observations and $D$ categorical variables. We show the $d$-th categorical variable in the $n$-th observation by $y_{nd}.$ Each categorical variable $y_{nd}$ can take values, for example, from 1 to $K$. Each of them are samples from their corresponding probability mass function with weights $\mathbf{f}_{nd}=(f_{nd1},\ldots,f_{ndK})$. We assume each random variable $f_{ndk}$ is a nonlinear function of the input variable $\mathbf{x}_{n}\in\mathbb{R}^{Q}$, where $Q$ is the dimension of $\mathbf{x}_{n}.$ Therefore, $f_{ndk}=\mathcal{F}_{dk}\left(\mathbf{x}_{n}\right).$

Since $\mathbf{x}$ and $\mathcal{F}$ are latent we need to assign prior distributions for them. We assume a Gaussian distribution prior with standard deviation $\sigma_{x}^{2}$ for the latent variables $\mathbf{x}$, and a Gaussian process (GP) prior for each of the functions $\mathcal{F}$. We also consider a set of $M$ variational inducing points. For each vector of latent function values $\mathbf{f}_{dk}$, we introduce a separate set of $M$ inducing variables $\mathbf{u}_{dk}$, evaluated at a set of inducing input locations given by $Z$. It is assumed that all $\mathbf{u}_{dk}$s are evaluated at the same inducing locations. The inducing variables are function points drawn from the GP prior and lie in same latent space as the $F$ variables (Fig. LABEL:fig:_FU). The generative model can be summarized as

	$\displaystyle{\mathbf{x}}_{nq}$	$\displaystyle\stackrel{{\scriptstyle\text{ iid }}}{{\sim}}\mathcal{N}\left(0,\sigma^{2}_{x}\right),$		(1)
	$\displaystyle\mathcal{F}_{dk}$	$\displaystyle\stackrel{{\scriptstyle\text{ iid }}}{{\sim}}\mathcal{GP}\left(0,\mathbf{K}_{d}\right),$		(2)
	$\displaystyle f_{ndk}$	$\displaystyle=\mathcal{F}_{dk}\left(\mathbf{x}_{n}\right),$		(3)
	$\displaystyle u_{mdk}$	$\displaystyle=\mathcal{F}_{dk}\left(\mathbf{z}_{m}\right),$		(4)
	$\displaystyle p(y_{nd}=k)$	$\displaystyle=\frac{\exp\left(f_{ndk}\right)}{\sum_{k^{\prime}=1}^{K}\exp\left(f_{ndk^{\prime}}\right)}.$		(5)

The marginal log-likelihood is intractable because of the covariance function of the GP and the Softmax likelihood. We will consider a variational approximation to the posterior distribution of $\mathbf{X},\mathbf{F}$ and $\mathbf{U}$ factorized as,

$$q(\mathbf{X},\mathbf{F},\mathbf{U})=q(\mathbf{X})q(\mathbf{U})p(\mathbf{F}|\mathbf{X},\mathbf{U}).$$

(6)

By applying Jensen’s inequality, we can write a lower bound of the log-evidence (ELBO) as

$$\log p(\mathbf{Y})=\log\int p(\mathbf{X})p(\mathbf{U})p(\mathbf{F}|\mathbf{X},\mathbf{U})p(\mathbf{Y}|\mathbf{F})\mathrm{d}\mathbf{X}\mathrm{d}\mathbf{F}\mathrm{d}\mathbf{U}\\ \geq-\mathrm{KL}(q(\mathbf{X})\|p(\mathbf{X}))-\mathrm{KL}(q(\mathbf{U})\|p(\mathbf{U}))\\ \quad\quad+\sum_{n=1}^{N}\sum_{d=1}^{D}\int q\left(\mathbf{x}_{n}\right)q\left(\mathbf{U}_{d}\right)p\left(\mathbf{f}_{nd}|\mathbf{x}_{n},\mathbf{U}_{d}\right)\\ \cdot\log p\left(\mathbf{y}_{nd}|\mathbf{f}_{nd}\right)\mathrm{d}\mathbf{x}_{n}\mathrm{d}\mathbf{f}_{nd}\mathbf{U}_{d}:=\mathcal{L},$$

(7)

where,

$$p\left(\mathbf{f}_{nd}|\mathbf{x}_{n},\mathbf{U}_{d}\right)=\prod_{k=1}^{K}\mathcal{N}(\mathbf{K}_{d,nM}\mathbf{K}_{d,MM}^{-1}\mathbf{u}_{dk},\\ K_{d,nn}-\mathbf{K}_{d,nM}\mathbf{K}_{d,MM}^{-1}\mathbf{K}_{d,Mn}).$$

(8)

The lower bound is still intractable because of the softmax likelihood, $\log p\left(\mathbf{y}_{nd}\mid\mathbf{f}_{nd}\right)$. Therefore, we will compute the lower bound $\mathcal{L}$ and its derivatives with the Monte Carlo method. We draw samples of $\mathbf{x}_{n},\mathbf{U}_{d}$ and $\mathbf{f}_{nd}$ from $q\left(\mathbf{x}_{n}\right),q\left(\mathbf{U}_{d}\right),$ and $p\left(\mathbf{f}_{nd}\mid\mathbf{x}_{n},\mathbf{U}_{d}\right)$ respectively and estimate $\mathcal{L}$ with the sample average. We consider mean field variational approximation for the latent points $q(\mathbf{X})$ and a joint Gaussian distribution for $q(\mathbf{U})$ as,

$$q(\mathbf{U})=\prod_{d=1}^{D}\prod_{k=1}^{K}\mathcal{N}\left(\mathbf{u}_{dk}|\boldsymbol{\mu}_{dk},\mathbf{\Sigma}_{d}\right),$$

(9)

$$q(\mathbf{X})=\prod_{n=1}^{N}\prod_{q=1}^{Q}\mathcal{N}\left(x_{nq}|m_{nq},s_{nq}^{2}\right),$$

(10)

where the covariance matrix $\Sigma_{d}$ is shared for the same categorical variable $d$. The KL divergence in $\mathcal{L}$ can be computed analytically with the given variational distributions. We need to optimize the hyperparameters of each GP (parameters of $\mathbf{K}_{d}$), parameters of the variational random variables $\mathbf{u}_{dk}$, $\boldsymbol{\mu}_{dk}$, $\mathbf{\Sigma}_{d}$, mean $m_{nq}$ and variance $\sigma^{2}_{nq}$ of the latent inputs. The graphical model of the variational distributions for the inference part is shown in Fig. LABEL:graph2. For further details about the inference and learning hyperparameters refer to [gal2015latent].

In this section, we present our results with synthetic data. We first generated a dataset $\mathbf{Y}$ for $N=100$ patients and 10 categorical variables as shown in Fig. LABEL:graph1. These patients belonged to two clusters. We assumed each categorical variable can take values 0 and 1. In this way, we only needed to estimate the probability of one category, since $\operatorname{Pr}\{\mathrm{k}=0\}=1-\operatorname{Pr}\{\mathrm{k}=1\}$. The probability of the $d$-th category to be 1 for patient $x_{n}$ was proportional to the output of the function $f_{d}(x_{n})$, Fig. 1 (a). We generated samples $x_{n}$ from a mixture of two Gaussian distributions in order to see if the model could detect the underlying cluster of patients. These two clusters are illustrated in Fig. 1 (a). Note that the dimension of the latent space was one. However, we initialized the dimension at a higher value and let the model learn the lower dimension of the latent space. Figure 1 (b) shows the inferred latent space for the first two dimensions. Figure 1 (c) plotted the train error. We can infer the dimensionality of the latent space by comparing the maximized ELBO for different initialization of $Q$.

We used data collected at SBUH of test-positive COVID-19 pregnant women. The dataset was composed of categorical data of 89 patients. All the variables are listed in Table 1. The COVID-19 patients may carry severe or mild symptoms, and some of them end up in an ICU. There have been studies that utilized ML algorithms to build a detection model for the severeness of COVID-19. The authors in [yao2020severity] have investigated the binary classification problem between severe and mild cases. The problem with these methods is that they do not provide uncertainty about the prediction. Moreover, they need subjective definition of features and outcomes. Most of the time, disease severity levels are more than just described by two classes. Thus, it is important to study a range of classes of disease severeness, from mild, such as common cough, various in-between levels of disease, to harsh that entail ending up in ICUs. This is why it is important to employ methods like categorical latent GPs to decide on their own on the number of categories of patients and based on the patterns they discover in the patients’ data.

In this analysis, we mapped the patient population into a two-dimensional space using only the symptoms at the time of diagnosis. Then we labeled the points by severity of the outcomes. The latent space representing outcomes is shown in Fig. 2. By looking at the mapped data, we can easily see that there is a population of patients at $Q_{1}=0.1$ most of which did not need hospitalization. Instead, all the patients that have been admitted to the hospital are concentrated at the upper left corner of the space. Among them, three patients were admitted to ICU after developing severe outcomes. Here we have also observed one patient who is an outlier because that patient has been in ICU, and in the latent space she was lying in the far right of the space. Note that the pregnant women under investigation could have been in the hospital for reasons other than COVID. Many of them tested positive when they were completely asymptomatic.

Finally we mapped patients using all the categorical variables from Table 1 into a two-dimensional latent space. The latent points and the clustering of the patients are shown in Fig. 3. The intensity of green color in the figure reflects the density of the population in the latent space. The highest density is around the rightmost group of patients. One might argue that there are three clusters of patients, of which the one in the middle is the least concentrated. These clusters allow us to study the relationships among the patients and in particular, understand why patients grouped in a cluster are similar to other patients in the same cluster.