A Joint Learning Approach for Semi-supervised Neural Topic Modeling

Most past work in neural topic models focused on designing inference networks with better model specification in the unsupervised setting. One line of recent research attempts to improve topic model performance by modifying the inference network through changes to the topic priors or regularization over the latent space Miao et al. (2016); Srivastava and Sutton (2017); Nan et al. (2019). Another line of research looks towards incorporating the expressivity of word embeddings to topic models Dieng et al. (2019a, b).

In contrast to existing work on neural topic models, our approach does not mainly focus on model specification; rather, we create a broader architecture into which neural topic models of all specifications can be trained in an upstream, semi-supervised setting. We believe that our architecture will enable existing neural topic models to be used in a wider range of real-word scenarios where we leverage labeled data alongside unlabeled data and use the knowledge present in document labels to further supervise topic models. Moreover, by directly tying our topic distributions to the labels through label-indexing, we create topics that are specific to labels, making these topics more interpretable as users are directly able to glean what types of documents each of the topics are summarizing.

Most supervised topic models follow the downstream supervised framework introduced in s-LDA Blei and McAuliffe (2010). This framework assumes a two-stage setting in which a topic model is trained and then a predictive model for the document labels is trained independently of the topic model. Neural topic models following this framework have also been developed, with the predictive model being a discriminative layer attached to the learned topics, essentially treating topic modeling as a dimensionality reduction tool Wang and Yang (2020); Cao et al. (2015); Huh and Fienberg (2012).

In contrast to existing work, LI-NTM is an upstream generative model (Figure 2, Figure 3) following a prediction-constrained framework. The upstream setting allows us to implicitly train our classifier and topic model in a one-stage setting that is end-to-end. This has the benefit of allowing us to tune the trade-off between our classifier and topic model performance in a prediction-constrained framework, which has been shown to achieve better empirical results when latent variable models are used as a dimensionality reduction tool Hughes et al. (2018); Hope et al. ; Sharma et al. (2021). Furthermore, the upstream setting allows us to introduce the document label classifier as a latent variable, enabling our model to work in semi-supervised settings.

LDA is a probabilistic generative model for topic modeling Blei et al. (2003); Blei and McAuliffe (2010). Through the process of estimation and inference, LDA learns K topics $\beta_{1:K}$. The generative process of LDA posits that each document is a mixture of topics with the topics being global to the entire corpus. For each document, the generative process is listed below:

1. 1.

   Draw topic proportions $\theta_{d}\sim$ Dirichlet$(\alpha_{\theta})$

2. 2.

   For each word $w$ in document:

   1. (a)

      Draw topic assignment $z_{dn}\sim$ Cat($\theta_{d}$)

   2. (b)

      Draw word $w_{dn}\sim$ Cat($\beta{z_{dn}}$)

3. 3.

   Draw responses $y|z_{1:N},\eta,\sigma^{2}\sim\mathcal{N}(\eta^{T}\bar{z},\sigma^{2})$ (if supervised)

where $\bar{z}:=\frac{1}{N}\sum_{i=1}^{N}z_{n}$ and the parameters $\eta,\sigma^{2}$ are estimated during inference. $\alpha_{\theta}$ is a hyper-parameter that serves as a prior for topic mixture proportions. In addition we also have hyper-parameter $\alpha_{\beta}$ that we use to place a dirichlet prior on our topics, $\beta_{k}\sim$ Dirichlet$(\alpha_{\beta})$.

Deep Generative Models serve as the bridge between probabilistic models and neural networks. Specifically, deep generative models treat the parameters of distributions within probabilistic models as outputs of neural networks. Deep generative models fundamentally work because of the re-parameterization trick that allows for backpropogation through Monte-Carlo samples of distributions from the location-scale family. Specifically, for any distribution $g(\cdot)$ from the location-scale family, we have that

thus allowing differentiation with respect to $\mu,\sigma^{2}$.

The Variational Auto-encoder is the simplest deep generative model Kingma and Welling (2014) and it’s generative process is as follows:

where $\mu_{\theta}(z),\Sigma_{\theta}(z)$ are both parameterized by neural networks with variational parameters $\theta$. Inference on a variational autoencoder is done through approximating the true posterior $p(z|x)$ which is often intractable with an approximation $q_{\phi}(z|x)$ that is parametrized by a neural network.

The M2 model is the semi-supervised extension of the variational auto-encoder where the input is modeled as being generated by both a continuous latent variable $z$ and the class label $y$ as a latent variable Kingma et al. (2014). It follows the generative process below:

where $\pi$ is parameterizing the distribution on $y$ and $\mu_{\theta}(y,z),\Sigma_{\theta}(y,z)$ are both parameterized by neural networks. We then approximate the true posterior $p(y,z|x)$ using by saying

where $q_{\phi}(y|x)$ is a classifier that’s used in the un-labeled case and $q_{\phi}(z|y,x)$ is a neural network that takes in the true labels if available and the outputted labels from $q_{\phi}(y|x)$ if unavailable.

We begin first by looking at a family of variational distributions $q_{\phi}(\delta_{d}|x_{d})$ in modeling the untransformed topic proportions and $q_{\nu}(y_{d}|x_{d})$ in modeling the classifier. More specifically, $q_{\phi}(\delta_{d}|x_{d})$ is a Gaussian whose mean and variance are parameterized by neural networks with parameter $\phi$ and $q_{\nu}(y_{d}|x_{d})$ is a distribution over the labels parameterized by a MLP with parameter $\nu$ Kingma and Welling (2014); Kingma et al. (2014).

We use this family of variational distributions alongside our classifier to lower-bound the marginal likelihood. The evidence lower bound (ELBO) is a function of model and variational parameters and provides a lower bound for the complete data log-likelihood. We derive two ELBO-based loss functions: one for the labeled case and one for the unlabeled case and we compute a linear interpolation of the two for our overall loss function.

where Equation 1 serves as our unlabeled loss and Equation 2 serves as our labeled loss. $\mathcal{H}(\cdot,\cdot)$ is the cross-entropy function. $\tau$ and $\rho$ are hyper-parameters on the KL and cross-entropy terms in the loss respectively.

These hyper-parameters are well motivated. $\tau$ is seen to be a hyper-parameter that tempers our posterior distribution over weights, which has been well-studied and shown to increase robustness to model mis-specification Mandt et al. (2016); Wenzel et al. (2020). Lower values $\tau$ would result in posterior distributions with higher probability densities around the modes of the posterior. Furthermore, the $\rho$ hyperparameter in our unlabeled loss is the core hyperparameter that makes our model fit the prediction-constrained framework, essentially allowing us to trade-off the between classifier and topic modeling performance Hughes et al. (2018). Increasing values of $\rho$ corresponds to emphasizing classifier performance over topic modeling performance.

We treat our overall loss as a combination of our labeled and unlabeled loss with $\lambda\in(0,1)$ being a hyper-parameter weighing the labeled and unlabeled loss. $\lambda$ allows us weigh how heavily we want our unlabeled data to influence our models. Example cases where we may want high values of $\lambda$ are when we have poor classifier performance or a disproportionate amount of unlabeled data compared to label data, causing the unlabeled loss to completely outweigh the labeled loss.

We optimize our loss with respect to both the model and variational parameters and leverage the reparameterization trick to perform stochastic optimization Kingma and Welling (2014). The training procedure is shown in Algorithm 1 and a visualization of a forward pass is given in Figure 3. This loss function allows us to jointly learn our classification and topic modeling elements and we hypothesize that the implicit regularization from joint learning will increase performance for both elements as seen in previous research studies Zweig and Weinshall (2013).

We compare our topic model to the Embedded Topic Model (ETM), which is the current state of the art neural topic model that leverages word embeddings alongside variational autoencoders for unsupervised topic modeling Dieng et al. (2019a). Further details about ETM are shown in the appendix (subsection A.2). Furthermore, our baseline for our jointly trained classifier is a classifier with the same architecture outside of our jointly trained setting.

We constructed our synthetic data to evaluate LI-NTM in ideal and worst-case settings.

• •

  Ideal Setting: An ideal setting for LI-NTM consists of a corpus with similar word distributions for documents with the same label and very dissimilar word distributions for documents with different labels

• •

  Worst Case Setting worst-case setting for LI-NTM consists of a corpus where the label has little to no correlation with the distribution of words in a document.

Since the labels are a fundamental aspect of LI-NTM we wanted to investigate how robust LI-NTM is in a real-word setting, specifically looking at how robust it was to certain types of mis-labeled data points. By jointly training our classifier with our topic model, we hope that by properly trading off topic quality and classification quality, our model will be more robust to mis-labeled data since we are able to manually tune how much we want to depend on the data labels.

We use the same distributions to generate the documents for both the ideal and worst-case data. In particular, we consider a vocabulary with $V=20$ words, and a task with $L=2$ labels. Documents are generated from one of two distributions, $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$. $\mathcal{D}_{1}$ generates documents which have many occurrences of the first 10 words in the vocabulary (and very few occurrences of the last 10 words), while $\mathcal{D}_{2}$ does the opposite, generating documents which have many occurrences of the last 10 words in the vocabulary (and very few occurrences of the first 10 words). The distributions $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$ have parameters which are generated randomly for each trial, although the shape of the distributions is largely the same from trial to trial.

In the ideal case, the label corresponds directly to the distribution from which the document was generated. For the worst-case data, the label is 0 if the number of words in the document is an even number, and 1 otherwise, ensuring there is little to no correlation between label and word distributions in a document. Note that in our synthetic data experiments, all of the data is labeled. The effectiveness of LI-NTM in semi-supervised domains is evaluated in our AG News experiments.

The AG News dataset is a collection of news articles collected from more than 2,000 news sources by ComeToMyHead, an academic news search engine. This dataset includes 118,000 training samples and 7,600 test samples. Each sample is a short text with a single four-class label (one of world, business, sports and science/technology).

To evaluate our models, we used accuracy as a metric to gauge the quality of the classifier and perplexity to gauge the quality of the model as a whole. We opted to use perplexity as it is a measure for how well the model generalizes to unseen test data.

Takeaway: More topics lead to better performance, especially when the label is uninformative.

First, we note that as we increase the number of topics, the performance of LI-NTM on ideal case labels, LI-NTM on worst case labels, and ETM improves as shown in Table 1. This is expected as having more topics gives the model the capacity to learn more diverse topic-word distributions which leads to an improved reconstruction. However, we note that LI-NTM trained on the worst-case labels benefits most from the increase in the number of topics.

Takeaway: Label Indexing is highly effective when labels partition the dataset well.

Next, we note that LI-NTM trained on the ideal case label synthetic dataset outperforms ETM with respect to perplexity (see Table 1). This result can be attributed to the fact that LI-NTM leverages label indexing to learn the label-topic-word distribution. Since the ideal case label version of the dataset was constructed such that the label strongly partitions the dataset into two groups (each of which has a very distinct topic-word distribution), and since we had perfect classifier accuracy (the ideal case label dataset was constructed such that the classification problem was trivial), LI-NTM is able to use the output from the classifier to index into the topic-word distribution with 100% accuracy.

If we denote the topic-word distribution corresponding to label $0$ by $\beta_{0}$ and the topic-word distribution corresponding to label $1$ by $\beta_{1}$, we note that LI-NTM is able to leverage $\beta_{0}$ to specialize in generating the words for the documents corresponding to label $0$ while using $\beta_{1}$ to specialize in generating the words for the documents corresponding to label $1$ (see Figure 4). Overall, this result suggests that LI-NTM performs well in settings when the dataset exhibits strong label partitioning.

Takeaway: With proper hyperparameters, LI-NTM is able to achieve good topic model performance even when we have uninformative labels.

We now move to examining the results produced by LI-NTM trained on the worst-case labels. In this data setting, we investigated the robustness of the LI-NTM architecture. Specifically, we looked at a worst-case dataset, where we have labels that are uninformative and are thus not good at partitioning the dataset into subsets that have distinct topics.

In the worst-case setting, we define the following two instances of the LI-NTM model.

• •

  LI-NTM (V1) This model refers to the normal ($\rho\neq 0$) version of the model trained in the worst case setting.

• •

  LI-NTM (V2) This model refers to a LI-NTM model with zero-ed out classification loss ($\rho=0$), essentially pushing the model to only accurately reconstruct the original data.

For LI-NTM (V1), we did see decreases in performance; namely, that V1 has a worse perplexity than both ETM and ideal case LI-NTM. This aligns with our expectation that having a label with very low correlation to the topic-word distributions in the document results in poor performance in LI-NTM. This can be attributed to the failure of LI-NTM to adequately label-index in cases where this occurs.

However, for LI-NTM (V2) we found that we were actually able to achieve lower perplexity than ETM when the model was told to produce more than 2 topics, even with uninformative labels. To understand why this was happening, we analyzed the accuracy of the original classifier in LI-NTM (V2) on both the worst-case labels (which it was trained on) and the ideal-case labels (which it was not trained on). We report our results in Table 2. The key takeaway is that we observed a much higher accuracy on the ideal labels compared to the worst-case labels. This suggests that when $\rho=0$ the classifier implicitly learns the ideal labels that are necessary to learn a good reconstruction of the data, even when the provided labels are heavily uninformative or misspecified. This shows the benefit of label-indexing and of jointly learning our topic model and classifier in a semi-supervised fashion. Even in cases with uninformative data points, by setting $\rho=0$, the joint learning setting of our classifier and topic model pushes the classifier, through the need for successful document reconstruction, to generate a probability distribution over labels that is close to the true, ideal-case labels despite only being given uninformative or mis-labeled data.

Takeaway: Combining label-indexing with semi-supervised learning increases topic model performance.

In Table 3 we observe that perplexity decreases as the model sees more unlabeled data. We also note that LI-NTM has a lower perplexity than ETM in higher data settings, supporting the hypothesis that guiding the reconstruction of a document exclusively via label-specific topics makes reconstruction an easier task. In the lowest data regime (5% labeled, 5% unlabeled), LI-NTM performs worse than ETM. This suggests that while in high-data settings, LI-NTM is able to effectively leverage $L=4$ sets of topics, in low-data settings there are not enough documents to learn sufficient structure.

Takeaway: Topic modeling supervises the classifier, resulting in better classification performance.

Jointly learning the classifier and topic model also seem to benefit the classifier; Table 3 shows classification performance increases linearly with the amount of unlabeled data. The accuracy increase suggest the task of reconstructing the bag of words is helpful in news article classification.

Select topics learned from LI-NTM on the AG News Dataset are presented in Table 4 and the distributions are visualized in the appendix Figure A1.