Variational Gaussian Topic Model with Invertible Neural Projections

Distributed semantic models (i.e. word embeddings) have recently been applied successfully in many NLP tasks.

Neural based models, such as word2vec, have been more efficient thanks to the skip-gram with a negative sampling training method Mikolov et al. (2013). To solve the limitation that word2vec only employs local context information, Pennington Pennington et al. (2014) proposed GloVe, a global log-bilinear regression model, which combines the advantages of the global matrix factorization and local context window methods. To generate a vector for an out of vocabulary word, FastText Joulin et al. (2017); Athiwaratkun et al. (2018), has been proposed. It treats each word as made of character n-grams and word vectors are then computed from the sum of their n-gram representations.

Invertible Projection Learning, also known as generative flows, is first described in NICE Dinh et al. (2014) and has recently received much attention Dinh et al. (2016); Jacobsen et al. (2018); Kingma and Dhariwal (2018).

Typically, generative flows have been proposed for image generation. NICE Dinh et al. (2014) firstly designed a composition of additive coupling layers to non-linearly project a complex high-dimensional densities into a simple prior and the inverse projection is used for generating images. To address the issue that variational auto-encoder could not approximate complex posterior, Danilo Rezende and Mohamed (2015) incorporated the normalizing flow into variational inference for specifying arbitrarily complex posterior distributions. Also, He He et al. (2018) incorporated the generative flow into the Hidden Markov Model for unsupervised part-of-speech tagging.

To overcome the difficult exact inference of topic models based on the directed graph, a replicated softmax model, called RSM, based on the Restricted Boltzmann Machines was proposed in  Hinton and Salakhutdinov (2009). Inspired by the variational auto-encoder, Miao et al. Miao et al. (2016) used the multivariate Gaussian as the prior distribution of latent space and proposed the Neural Variational Document Model (NVDM) for text modeling. Recently, to deal with the inappropriate Gaussian prior of NVDM, Srivastava and Sutton Srivastava and Sutton (2017) proposed the NVLDA and ProdLDA which approximates the Dirichlet prior using a logistic normal distribution. Furthermore, Dieng Dieng et al. (2019) proposed the ETM based on VAE which models topics in embedding space.

Although the extensive exploration of the above fields, scarce work has been done to incorporate word representation learning and invertible projection learning into neural topic modeling. Thus, we propose two novel topic modeling approaches, called VaGTM and VaGTM-IP, which differs published works in the following facets: (1). Unlike the NVDM, NVLDA, ProdLDA and ATM which only uses word co-occurrence information, our proposed VaGTM models each topic with a multivariate Gaussian distribution which could incorporate the word semantic relatedness into the modeling process; (2). To deal with the issue that word embeddings of topic-associated words do not follow a multivariate Gaussian, we incorporate a flow-based projector into the modeling process and propose the VaGTM-IP .

To obtain interpretable topics, Dirichlet prior over topics is commonly used Wallach et al. (2009). However, it is difficult to explicitly model topics with Dirichlet prior since it is hard to develop an effective reparameterization. Thus, we solve the issue by constructing a Laplace approximation Hennig et al. (2012).

Aiming at compressing the weighted bag of words representation $\vec{X}$ into the latent topic distribution $\vec{\theta}$, the encoder $q_{\bm{\psi}}(\vec{\theta}|\vec{X})$, parameterized with $\bm{\psi}$, contains five layers which are one $V$-dimensional document representation layer, two $H$-dimensional semantic extraction layers, one $K$-dimensional reparameterization layer and one $K$-dimensional topic distribution layer as shown in Fig. 1. Firstly, it projects $\vec{X}$ into an $H$-dimensional semantic space through semantic extraction layers based on the transformation:

	$\displaystyle\vec{a}_{s}^{1}$	$\displaystyle=\ln(1+\exp(W_{s}^{1}\vec{X}+\vec{b}_{s}^{1}))$		(1)
	$\displaystyle\vec{a}_{s}^{2}$	$\displaystyle=\ln(1+\exp(W_{s}^{2}\vec{a}_{s}^{1}+\vec{b}_{s}^{2}))$		(2)

where $W_{s}^{1}\in\mathbb{R}^{H\times V}$ and $W_{s}^{2}\in\mathbb{R}^{H\times H}$ are weight matrices, $\vec{b}_{s}^{1}$ and $\vec{b}_{s}^{2}$ are corresponding basis terms, $\vec{a}_{s}^{1}$ and $\vec{a}_{s}^{2}$ are the semantic representations of $\vec{X}$, $\exp(\cdot)$ is the element-wise exponential function.

To further infer the posterior distribution over topics, following the Laplace approximation, a Gaussian distribution should be generated. Thus, the encoder projects $\vec{a}_{s}^{2}$ into two $K$-dimensional Gaussian parameters $\vec{\mu}$ and $\vec{\sigma^{2}}$ using:

	$\displaystyle\vec{\mu}$	$\displaystyle=\textrm{BN}(W_{r}^{\mu}\vec{a}_{s}^{2}+\vec{b}_{\mu})$		(3)
	$\displaystyle\vec{\sigma^{2}}$	$\displaystyle=\exp(\textrm{BN}(W_{r}^{\sigma}\vec{a}_{s}^{2}+\vec{b}_{\sigma}))$		(4)

where $W_{r}^{\mu}\in\mathbb{R}^{K\times H}$, $W_{r}^{\sigma}\in\mathbb{R}^{K\times H}$, $\vec{b}_{\mu}$ and $\vec{b}_{\sigma}$ are weight matrices and basis terms of reparameterization layers, $\textrm{BN}(\cdot)$ is batch normalization. And $\vec{\mu}$ and $\vec{\sigma^{2}}$ are mean and diagonal covariance of posterior Gaussian corresponding to input $\vec{X}$.

Finally, based on the reparameterization trick Kingma and Welling (2013) and Laplace approximation Hennig et al. (2012), $\vec{X}$ could be inferred to a $K$-dimensional topic distribution $\vec{\theta}$ using below transformation:

	$\displaystyle\vec{z}$	$\displaystyle=\vec{\mu}+\vec{\varepsilon}\odot\vec{\sigma},\vec{\varepsilon}\sim\mathcal{N}(\vec{0},\emph{I})$		(5)
	$\displaystyle\vec{\theta}$	$\displaystyle=\text{softmax}(\vec{z})$		(6)

Here, $\mathcal{N}(\vec{0},\emph{I})$ is $K$-dimensional standard multivariate Gaussian, and the posterior distribution $q(\vec{\theta}|\vec{X})$ follows the Dirichlet distribution in softmax basis MacKay (1998).

The decoder $p_{\bm{\omega}}(\vec{X}^{\prime}|\vec{\theta})$, parameterized with $\bm{\omega}$, reconstructs the documents by independently generating the words ($\vec{\theta}\rightarrow\vec{X}^{\prime}_{i}$). Besides, to incorporate the word relatedness captured in word embeddings, VaGTM models each topic with a multivariate Gaussian as depicted in the left panel of Fig. 1.

Concretely, VaGTM employs multivariate Gaussian $\mathcal{N}(\vec{\mu}_{k},\Sigma_{k})$ to model the $k$-th topic. where $\vec{\mu}_{k}$ and $\Sigma_{k}$ represent mean and covariance matrix. Following the probability density of Gaussian, for each word $v\in\{1,2,...,V\}$, its probability in the $k$-th topic $\phi_{k,v}$ could be calculated as:

	$\displaystyle p(\vec{w}_{v}|$	$\displaystyle topic=k)=\mathcal{N}(\vec{w}_{v};\vec{\mu}_{k},\Sigma_{k})$
	$\displaystyle=$	$\displaystyle\frac{\exp(-\frac{1}{2}(\vec{w}_{v}-\vec{\mu}_{k})^{\text{T}}\Sigma^{-1}(\vec{w}_{v}-\vec{\mu}_{k}))}{\sqrt{(2\pi)^{D_{w}}|\Sigma_{k}|}}$		(7)
	$\displaystyle\phi_{k,v}=$	$\displaystyle\frac{p(\vec{w}_{v}|topic=k)}{\sum_{v=1}^{V}p(\vec{w}_{v}|topic=k)}$		(8)

where $\vec{w}_{v}$ is the word embedding of word $v$, $V$ is the vocabulary size, $|\Sigma_{k}|=\rm det$ $\Sigma_{k}$ represents the determinant of covariance matrix $\Sigma_{k}$, $D_{w}$ is the dimension of word embeddings, $\vec{\phi}_{k}$ is the normalized word distribution of the $k$-th topic.

To reconstruct the document $\vec{X}$, topic-word distributions and the encoded topic distribution $\vec{\theta}$ are combined using:

$\displaystyle p_{rec}(\vec{v}|\vec{\theta})=\sum_{k=1}^{K}\vec{\phi}_{k}\cdot\theta_{k}$

(9)

where $K$ denotes the topic number, $\theta_{k}$ means the proportion of $k$-th topic in $\vec{X}$, and $p_{rec}(v|\vec{\theta})$ is the conditional distribution over the vocabulary which is employed to reconstruct the $\vec{X}$ by maximizing the equation:

$$p(\vec{X}^{\prime}|\vec{\theta})=\sum_{v=1}^{V}X_{v}\cdot\log\left[p_{rec}(v|\vec{\theta})\right]$$

(10)

where $X_{v}$ is the tf-idf weight of the $v$-th word in document $\vec{X}$, and $\vec{X}^{\prime}$ is the reconstructed document.

Word embeddings indeed encode numerous semantic regularities. However, widely used word representation scheme, such as word2vec Mikolov et al. (2013) and Glove Pennington et al. (2014), do not model embeddings with Gaussian and the resulting embeddings often follow a complicated distribution $\mathbb{P}_{w}$. Thus, it is not accurate to approximate $\mathbb{P}_{w}$ with mixed Gaussian $\mathbb{P}_{m}$ as VaGTM.

To approximate the $\mathbb{P}_{w}$ and yield a mixed Gaussian embedding space which is more suitable for topic modeling, a non-linear neural projector $f(\cdot)$ is employed to deterministically transform the mixed Gaussian embedding space to the pre-trained word embedding space. The vector representation of word $v$ in mixed Gaussian embedding space is denoted as $\vec{e}_{v}\in\mathbb{R}^{D_{e}}$, $D_{e}$ is the dimension of mixed Gaussian embedding space. Thus, for each word $v$, it has the transformation below:

$$\vec{w}_{v}=f(\vec{e}_{v})$$

(11)

Due to the reconstruction of $\vec{X}$ is conditioned on word embeddings, as shown in the right panel of Fig. 1, the projector $f(\cdot)$ should be invertible. To this end, a flow-based projector Dinh et al. (2014) is employed here to bridge the mixed Gaussian embedding space and word embedding space since it can transform a simple distribution into a complicated one.

Thus, decoding with invertible neural projection, the probability of word $v$ in the $k$-th topic is proportional to $p(\vec{w}_{v}|topic=k)$ which is defined as:

	$\displaystyle p($	$\displaystyle\vec{w}_{v}|topic=k)=\int p(\vec{w}_{v}|\vec{e}_{v})\cdot p(\vec{e}_{v}|topic=k)d\vec{e}_{v}$
		$\displaystyle=\int\square\cdot\mathcal{N}(f^{-1}(\vec{w}_{v});\vec{\mu}_{k},\Sigma_{k})\left|\mathrm{det}\frac{\partial f^{-1}}{\partial\vec{w}_{v}}\right|d\vec{w}_{v}$
		$\displaystyle=\mathcal{N}(f^{-1}(\vec{w}_{v});\vec{\mu}_{k},\Sigma_{k})\cdot\left|\mathrm{det}\frac{\partial f^{-1}}{\partial\vec{w}_{v}}\right|$		(12)

where $\frac{\partial f^{-1}}{\partial\vec{w}_{v}}$ represents the Jacobian matrix of inverse projection function $f^{-1}(\cdot)$ at $\vec{w}_{v}$ which is nonzero and differentiable if and only if $f^{-1}(\cdot)$ exists, and $\left|\mathrm{det}\frac{\partial f^{-1}}{\partial\vec{w}_{v}}\right|$ denotes the absolute value of its determinant. The symbol $\square$ in Eq. 12 represents $\delta(\vec{w}_{v}-f(\vec{e}_{v}))$. Here, $\delta(\cdot)$ is the Dirac delta function centered at $f(\vec{e}_{v})$ which is defined as:

$$p(\vec{w}_{v}|\vec{e}_{v})=\delta(\vec{w}_{v}-f(\vec{e}_{v}))=\left\{\begin{aligned} \infty&&\vec{w}_{v}=f(\vec{e}_{v})\\ 0&&\mathrm{otherwise}\end{aligned}\right.$$

(13)

Comparing Eq. 7 and Eq. 12, it should be observed that the addition of the Jacobian term will increase the difficulty of optimizing as it requires that all component functions be invertible and also requires storage of large Jacobian matrices. To address this issue, we use additive coupling layer suggested in Dinh et al. (2014) to guarantee a unit Jacobian determinant and the invertibility. To reconstruct the document $\vec{X}$, it could be observed from Eq. 12 that only $f^{-1}(\cdot)$ is required. Thus, we explicitly design the inverse projector $f^{-1}(\cdot)$ as Fig. 2.

To accomplish the transformation, word embedding $\vec{w}_{v}$ is first partitioned into two halves of dimensions, $\vec{w}^{1}_{v}$ and $\vec{w}^{2}_{v}$, respectively. And the first coupling layer is represented by the nonlinear transformation from $\vec{w}_{v}$ to $\vec{h}_{v}(1)$ which could be defined as:

$\displaystyle\vec{h}_{v}^{1}(1)=\vec{w}_{v}^{1},\quad\vec{h}_{v}^{2}(1)=\vec{w}_{v}^{2}+m(\vec{w}_{v}^{1})$

(14)

where $m(\cdot):\mathbb{R}^{D_{w}/2}\rightarrow\mathbb{R}^{D_{w}/2}$ is the coupling function which could be designed as any nonlinear function with same input and output shape. We choose additive coupling layer to reduce the computational consumption. To build a more complex transformation, we compose several coupling layers and exchange the role of two half vectors at each layer to ensure that the composition of two layers modifies every dimension.

Then, following the reconstruction procedure illustrated in decoding process of VaGTM (subsection 3.2), the document $\vec{X}$ could be reconstructed in a similar way in VaGTM-IP.

In variational Bayesian, the variational lower bound on the marginal likelihood of document $\vec{X}$ is usually formed as:

	$\displaystyle\mathcal{L}(\vec{X};\bm{\psi},\bm{\omega})=$
	$\displaystyle\mathbb{E}_{q_{\bm{\psi}(\vec{\theta}|\vec{X})}}\log p_{\bm{\omega}}(\vec{X}|\vec{\theta})-KL(q_{\bm{\psi}}(\vec{\theta}|\vec{X})\|p_{\omega}(\vec{\theta}))$		(15)

where $\vec{\theta}$ denotes the topic distribution of $\vec{X}$. To mimic the Dirichlet prior over $\vec{\theta}$, the Laplace approximation is used. and the variational lower bound is rewrited as:

	$\displaystyle\mathcal{L}(\vec{X};\bm{\psi},\bm{\omega}$	$\displaystyle)=\sum_{v=1}^{V}X_{v}\cdot\log\left[p_{rec}(v|\vec{\theta})\right]$
	$\displaystyle-\frac{1}{2}$	$\displaystyle\left[\mathrm{tr(\Sigma^{-1}\Sigma_{0})}+\Delta-K+\ln\frac{\mathrm{det}\ \Sigma}{\mathrm{det}\ \Sigma_{0}}\right]$		(16)

where $\Delta=(\vec{\mu}-\vec{\mu}_{0})^{\mathsf{T}}\Sigma^{-1}(\vec{\mu}-\vec{\mu}_{0})$, $\vec{\mu}_{0}$ and $\Sigma_{0}$ are means and covariance matrix of standard multivariate Gaussian, and $p_{rec}(v|\vec{\theta})$ is the $v$-th dimension of reconstruction distribution defined as Eq. 9.

To validate the effectiveness of proposed VaGTM and VaGTM-IP for topic modeling, 20Newsgroups ¹¹1http://qwone.com/ jason/20Newsgroups/ dataset, Grolier ²²2https://cs.nyu.edu/ roweis/data/ dataset and NYTimes ³³3http://archive.ics.uci.edu/ml/datasets/Bag+of+Words dataset are selected. Details are summarized below:

• •

  20Newsgroups dataset, provided in Lang (1995), it is a collection of approximately 20,000 newsgroup articles, partitioned evenly across 20 different newsgroups.

• •

  Grolier dataset is built from Grolier Multimedia Encycopedia, and its content covers almost all the fields in the world, such as religious, technology, economics and etc.

• •

  NYTimes dataset is a collection of news articles published between 1987 and 2007, and the dataset has a wide range of topics, such as sports, politics and etc.

Besides, we choose the following approaches as baselines:

• •

  LDA Blei et al. (2003), is a topic model that generates topics based on bag-of-words assumption. We implement the LDA model with the suggested configuration Griffiths and Steyvers (2004).

• •

  Gaussian-LDA Das et al. (2015), is a conventional topic model which uses the word embeddings information. The original implementation is used in this paper ⁴⁴4https://github.com/rajarshd/Gaussian LDA.

• •

  NVDM Miao et al. (2016) is an unsupervised text modeling approach based on variational auto-encoder. We use the original implementation in the paper ⁵⁵5https://github.com/ysmiao/nvdm.

• •

  NVLDA Srivastava and Sutton (2017), is a neural topic model based on VAE. It models topics with logistic normal prior, we use the original implementation ⁶⁶6https://github.com/akashgit/autoencoding vi for topic models.

• •

  ProdLDA Srivastava and Sutton (2017), is a variant of NVLDA, in which the distribution over individual words is a product of experts rather than the mixture model.

• •

  Scholar Card et al. (2018), is a neural topic modeling approach based on variational autoencoder, we use the original implementation ⁷⁷7https://github.com/dallascard/scholar.

• •

  ETM Dieng et al. (2019), is a neural topic model on embedding space, we use the original implementation⁸⁸8https://github.com/adjidieng/ETM.

• •

  ATM Wang et al. (2019), is a neural topic modeling approach based on adversarial training, we re-implement the ATM follow the default parameter settings.

For the NYTimes dataset, 100,000 articles are randomly selected, and low frequency words are removed here. The statistics of the processed datasets are listed in Table 1.

Typically, the likelihood of held-out documents and topic coherence metrics are used to evaluate topic modeling performance. However, as pointed out in Chang et al. (2009) the likelihood of held-out documents doesn’t correspond to human judgment, we follow Röder et al. (2015) and choose four topic coherence metrics which are C_P, C_A, NPMI and UCI to evaluate the extracted topics, and higher value implies more coherent topic. All the topic coherences are computed with the Palmetto⁹⁹9https://github.com/dice-group/Palmetto library and each topic is represented by top ten words according to the topic-word distribution.

To compare the topic extraction performance comprehensively, we firstly make a comparison of topic coherence vs. different topic proportions. In this part, we conduct the experiments on all datasets with five topic number settings [20, 30, 50, 75, 100]. Concretely, we calculate the average topic coherence among topics whose coherence are ranked at the top 70%, 80%, 90% and 100% positions. For example, to calculate the average NPMI value of VaGTM @80%, we should first compute the average NPMI coherence with the selected topics whose NPMI values are ranked at the top 80% for each topic number settings and then average the five averaged coherence values. The detailed comparison is shown in Fig. 3.

As the statistics shown in Fig. 3, both VaGTM and VaGTM-IP perform competitively compared with the baseline approaches. More detailed, VaGTM-IP achieves the highest values on four metrics (C_P, C_A, NPMI and UCI) among all topic proportions and datasets, and VaGTM outperforms the compared baselines as well except on C_A metric. For the 20Newsgroups dataset on C_A, though ProdLDA performs slightly better than VaGTM (70%, 80% and 90%), VaGTM obtains much higher topic coherence values considering C_P, NPMI and UCI metrics. Thus, it would be reasonable to conclude that both VaGTM and VaGTM-IP could generally generate more coherent topics than compared approaches. And the improved performance of VaGTM and VaGTM-IP may attribute to: 1). Both models incorporate the word relatedness into the modeling process which is helpful for grouping semantically related words into the same topic; 2) The invertible neural projector transforms word embeddings into a new embedding space and the transformed representation is more suitable for topic modeling.

Moreover, to explore how the performance varies with different topic numbers, we further compare the average topic coherence (considering all the extracted topics) vs. different topic number settings for each dataset. Fig. 4 depicts the detailed comparison. From the curves in the subplots, it is clear that the proposed approaches (VaGTM and VaGTM-IP) perform more stable than compared baselines. Also, in most cases, they obtain higher average coherence values. Similar to Fig. 3, ProdLDA performs well on C_A metric, and it gives slightly better results than VaGTM in 20 topic settings. But beyond that, VaGTM-IP and VaGTM largely outperforms all the other baselines on all other settings. This might attribute to the incorporation of the word relatedness contained in word embeddings. We also provide a numerical comparison of average coherence values in Table 2, each value is calculated by averaging the average topic coherences (considering all topics) over five topic number settings.

From the above topic coherence comparison (Fig. 3, Fig. 4 and Table 2), it is obvious that VaGTM and VaGTM-IP perform better than baselines. To validate this qualitatively, we provide four topic examples in Table 3 and out-of-topic words are highlighted in italic.

Besides, thanks to the usage of Gaussian distribution for topic modeling, the proposed VaGTM-IP could capture the semantic correlation between extracted topics. In detail, mean vector of learned topic-associated Gaussian could be viewed as topic embeddings in mixed Gaussian embedding space and hence the semantic relations between topics can be captured by the cosine similarity between mean vectors of topic associated Gaussian. And Fig. 5 show the visualization of topic correlation matrix for 20Newsgroups, Grolier and NYTimes datasets on 50 topic setting. Higher value implies that corresponding topics have higher semantic similarity.