Deep Clustering of Text Representations for Supervision-free Probing of Syntax

Contextualized text representations (Peters et al. 2018a; Devlin et al. 2019) have been used in many supervised NLP problems such as part-of-speech (POS) tagging (Tsai et al. 2019), syntactic parsing (Kitaev and Klein 2018; Zhou and Zhao 2019; Mrini et al. 2019), and coreference resolution (Lee, He, and Zettlemoyer 2018; Joshi et al. 2019; Wu et al. 2020), often leading to significant improvements. Recent works have shown that these representations encode linguistic information including POS (Belinkov et al. 2017), morphology (Peters et al. 2018a), and syntactic structure (Linzen, Dupoux, and Goldberg 2016; Peters et al. 2018b; Tenney, Das, and Pavlick 2019; Hewitt and Manning 2019).

While there has been a lot of focus on using contextualized representations in supervised settings for either solving NLP problems and interpreting these representations, the efficacy of these representations for unsupervised learning is not well explored¹¹1 Some recent work such as DIORA (Drozdov et al. 2019b, a) has explored specialized methods for unsupervised discovery and representation of constituents using ELMo (Peters et al. 2018a). (Jin et al. 2019) used ELMo with a normalizing flow model while (Cao, Kitaev, and Klein 2020) used RoBERTa (Liu et al. 2019b) for unsupervised constituency parsing.. Most of the recent work in “probing” contextual representations have focused on building supervised classifiers and using accuracy to interpret these representations. This has led to a debate as it is not clear if the supervised probe is probing the model or trying to solve the task (Hewitt and Manning 2019; Pimentel et al. 2020).

Thus, in this work, we explore a new clustering-based approach to probe contextualized text representations. Our probe allows for studying text representations with relatively less task-specific transformations due to the absence of supervision. Thus, our approach is arguably a less biased way to discover linguistic structure than supervised probes (Hewitt and Manning 2019; Pimentel et al. 2020; Zhou and Srikumar 2021).We focus on two syntactic formalisms: part-of-speech induction (POSI) and constituency labelling (CoLab), and explore the efficacy of contextualized representations towards encoding syntax in an unsupervised manner. We investigate the research question: Do contextualized representations encode enough information for unsupervised syntax induction? How do these perform on POSI, which has been traditionally solved using smaller context windows and morphology and span-based CoLab?

For both formalisms, we find that naively clustering text representations does not perform well. We speculate that this is because contextualized text representations are high-dimensional and not very friendly to existing clustering approaches. Thus, we develop a deep clustering approach (Xie, Girshick, and Farhadi 2016; Ghasedi Dizaji et al. 2017; Jiang et al. 2016; Chang et al. 2017; Yang, Parikh, and Batra 2016; Yang et al. 2017) which transforms these representations into a lower dimensional, clustering friendly latent space. This transformation is learnt jointly with the clustering using a combination of reconstruction and clustering objectives. The procedure iteratively refines the transformation and the clustering using an auxiliary target distribution derived from the current soft clustering. As this process is repeated, it gradually improves the transformed representations as well as the clustering. We show a t-SNE visualization of mBERT embeddings and embeddings learned by our deep clustering probe (SyntDEC) in Figure 1.

We further explore architectural variations such as pretrained subword embeddings from fastText (Joulin et al. 2017), a continuous bag of words (CBoW) loss (Mikolov et al. 2013), and span representations (Toshniwal et al. 2020) to incorporate task-dependent information into the latent space and observe sugnificant improvements. It is important to note that we do not claim that clustering contextualized representations is the optimal approach for POSI as representations with short context (Lin et al. 2015), (He, Neubig, and Berg-Kirkpatrick 2018) and word-based POSI (Yatbaz, Sert, and Yuret 2012) have shown best results. Our approach explores the potential of contextualized representations for unsupervised induction of syntax and acts as an unsupervised probe for interpreting these representations. Nevertheless, we report competitive many-to-one (M1) accuracies for POSI on the 45-tag Penn Treebank WSJ dataset as compared to specialized state-of-the-art approaches in the literature (He, Neubig, and Berg-Kirkpatrick 2018) and improve upon the state of the art on the 12 tag universal treebank dataset across multiple languages (Stratos, Collins, and Hsu 2016; Stratos 2019). We further show that our approach can be used in a zero-shot crosslingual setting where a model trained on one language can used for evaluation in another language. We observe impressive crosslingual POSI performance, showcasing the representational power of mBERT, especially when the languages are related. Our method also achieves competitive results on CoLab on the WSJ test set, outperforming the initial DIORA approach (Drozdov et al. 2019b) and performing comparably to recent DIORA variants (Drozdov et al. 2019a) which incorporate more complex methods such as latent chart parsing and discrete representation learning. In contrast to specialized state-of-the-art methods for syntax induction, our framework is more general as it demonstrates good performance for both CoLab and POSI by simply adapting the input representations.

We further investigate the effectiveness of multilingual BERT (mBERT) (Devlin et al. 2019) for POSI across multiple languages and CoLab in English and see improvement in performance by using mBERT for both tasks even in English. This is in contrast with the supervised experiments where both mBERT and E-BERT perform competitively. In contrast to various supervised probes in the literature (Liu et al. 2019a; Tenney, Das, and Pavlick 2019), our unsupervised probe finds that syntactic information is captured in higher layers on average than what was previously reported (Tenney, Das, and Pavlick 2019). Upon further layer-wise analysis of the two probes, we find that while supervised probes show that all layers of E-BERT contain syntactic information fairly uniformly, middle layers lead to a better performance on the investigated syntactic tasks with our unsupervised probe.

We consider two syntax induction problems in this work:

1. 1.

   Part-of-speech induction (POSI): determining part of speech of words in a sentence.

2. 2.

   Constituency label induction (CoLab): determining the constituency label for a given constituent (span of contiguous tokens).²²2Note that it is not necessary for constituents to be contiguous, but we only consider contiguous constituents for simplicity.

Figure 2 shows an illustration for the two tasks. In order to do well, both tasks require reasoning about the context. This motivates us to use contextualized representations, which have shown an ability to model such information effectively. Letting $[m]$ denote $\{1,2,\ldots,m\}$, we model unsupervised syntax induction as the task of learning a mapping function $C:X\longrightarrow[m]$. For POSI, $X$ is the set of word tokens in the corpus and $m$ is the number of part-of-speech tags.³³3$X$ is distinct from the corpus vocabulary; in POSI, we tag each word token in each sentence with a POS tag. For CoLab, $X$ is the set of constituents across all sentences in the corpus and $m$ is the number of constituent labels. For each element $x\in X$, let $c(x)$ denote the context of $x$ in the sentence containing $x$. The number $m$ of true clusters is assumed to be known. For CoLab, we also assume gold constituent spans from manually annotated constituency parse trees, focusing only on determining constituent labels, following Drozdov et al. (2019a).

We address unsupervised syntax induction via clustering, where $C$ defines a clustering of $X$ into $m$ clusters. We define a deep embedded clustering framework and modify it to support common NLP objectives such as continuous bag of words (Mikolov et al. 2013). Our framework jointly transforms the text representations into a lower-dimensions and learns the clustering parameters in an end-to-end setup.

Datasets: We evaluate our approach for POSI on two datasets: 45-tag Penn Treebank Wall Street Journal (WSJ) dataset (Marcus, Santorini, and Marcinkiewicz 1993) and multilingual 12-tag datasets drawn from the universal dependencies project (Nivre et al. 2016). The WSJ dataset has approximately one million words tagged with 45 part of speech tags. For multilingual experiments, we use the 12-tag universal treebank v2.0 dataset which consists of corpora from 10 languages.⁵⁵5We use v2.0 in order to compare to Stratos (2019). The words in this dataset have been tagged with 12 universal POS tags (McDonald et al. 2013). For CoLab, we follow the existing benchmark (Drozdov et al. 2019a) and evaluate on the WSJ test set. For POSI, as per the standard practice (Stratos 2019), we use the complete dataset (train + val + test) for training as well as evaluation. However, for CoLab, we use the train set to train our model and the test set for reporting results, following Drozdov et al. (2019a).

Yatbaz, Sert, and Yuret (2012) are not directly comparable to our work as they performed word-based POSI which attaches same tag to all the instances of the word, while all the other works in Table 1 perform token-based POSI. They use task-specific hand-engineered rules like presence of hyphen, apostrophe etc. which might not translate to multiple languages and tasks. (He, Neubig, and Berg-Kirkpatrick 2018) train a POSI specialized model with Markov syntax model and short-context word embeddings and report current SOTA on POSI. In contrast to their method, SyntDEC is fairly task agnostic.

Next, we leverage SyntDEC as an unsupervised probe to analyse where syntactic information is captured in the pretrained representations. Existing approaches to probing usually rely on supervised training of probes. However, as argued recently by (Zhou and Srikumar 2021), this can be unreliable. Our supervision-free probe arguably gets rid of any bias in interpretations due to the involvement of training data in probing.We compare our unsupervised probe to a reimplementation of the supervised shallow MLP based probe in Tenney, Das, and Pavlick (2019). Similar to their paper, we report Expected Layer under supervised and unsupervised settings for the two tasks in Figure 5. Expected Layer represents the average layer number in terms of incremental performance gains: ${E}_{\Delta}{[l]}=\frac{\sum_{l=1}^{L}l*\Delta^{(l)}}{\sum_{l=1}^{L}\Delta^{(l)}}$, where $\Delta^{(l)}$ is the change in the performance metric when adding layer $l$ to the previous layers. Layers are incrementally added from lower to higher layers. We use F1 and M1 score as the performance metric for supervised and unsupervised experiments respectively. We observe that:

1. 1.

   Expected Layer as per the unsupervised probe (blue) is higher than the supervised probe (green) for both tasks and models showing that unsupervised syntax induction benefits more from higher layers.

2. 2.

   There are larger differences between E-BERT and mBERT Expected Layer under unsupervised settings suggesting that our unsupervised probe utilizes mBERT and E-BERT layers differently than the supervised one.

Pires, Schlinger, and Garrette (2019); Wu and Dredze (2019) show that mBERT is effective at zero-shot crosslingual transfer. Inspired by this, we evaluate the crosslingual performance on 12-tag universal treebank (Table 4). The first row shows M1 accuracies when training and evaluating SyntDEC on the same language (monolingual). The second row shows M1 accuracies of the English-trained SyntDEC on other languages (crosslingual). In general, we find that clusters learned on a high-resource languages like English can be used for other languages. Similar to He et al. (2019), we use the distances of the languages with English to group languages as nearby or distant. The distance is calculated by accounting for syntactic, genetic, and geographic distances according to the URIEL linguistic database (Littell et al. 2017). Our results highlight the effectiveness of mBERT in crosslingual POSI. Even for Asian languages (ko, id, and ja), which have a higher distance from English, the performance is comparable across settings. For nearby languages, crosslingual SyntDEC performs well and even outperforms the monolingual setting for some languages.

In Table 5, we report the F1 and M1 score of constituency labelling (CoLab) over the WSJ test set. We represent constituents by concatenating embeddings of the first and last words in the span (where word embeddings are computed by averaging corresponding subword embeddings). We observe improvement over DIORA (Drozdov et al. 2019b), a recent unsupervised constituency parsing model, and achieve competitive results to recent variants that improve DIORA with discrete representation learning (Drozdov et al. 2019a). Our model and the DIORA variants use gold constituents for these experiments. We compute F1 metrics for comparing with previous work but also report M1 accuracies. As with POSI, our results suggest that mBERT outperforms both SpanBERT and E-BERT for the CoLab task as well. We also note that SpanBERT performs better than E-BERT, presumably because SpanBERT seeks to learn span representations explicitly. In the Appendix(Table 7), we explore other ways of representing constituents and note that mean/max pooling followed by clustering does not perform well. Compressing and finetuning the mean-pooled representation using SyntDEC (SyntDEC_Mean) is also suboptimal. We hypothesize that mean/max pooling results in a loss of information about word order in the constituent whereas the concatenation of first and last words retains this information. Even a stacked autoencoder (SAE) over the concatenation of first and last token achieves competitive results, but finetuning with SyntDEC improves the $F1_{\mu}$ by nearly 4.5%. This demonstrates that for CoLab also, the transformation to lower dimensions and finetuning to clustering friendly spaces is important for achieving competitive performance.

In this work, we explored the problem of clustering text representations for model interpretation and induction of syntax. We observed that off-the-shelf methods like KMeans are sub-optimal as these representations are high dimensional and, thus, not directly suitable for clustering. Thus, we proposed a deep clustering approach which jointly transforms these representations into a lower-dimensional cluster friendly space and clusters them. Upon integration of a small number of task-specific features, and use of multilingual representations, we find that our approach achieves competitive performance for unsupervised POSI and CoLab comparable to more complex methods in the literature. Finally, we also show that we can use the technique as a supervision-free approach to probe syntax in these representations and contrast our unsupervised probe with supervised ones.