Are Classes Clusters?

A lot of work has been done in this space, and there is not enough room here to go over all of it. So this summary only includes four of the most recent models in some detail (USE, SBERT, LASER, and DeCLUTR). It is omitting important ground work, in particular the following 3 studies and their resulting models:

• •

  SkipThought (Kiros et al., 2015): Basically “Thou shalt know a word by the company it keeps”, but extended to sentences. An unsupervised method based on the idea that sentences that are close to each other are more similar than sentences that are far away from each other, much like word2vec and glove.

• •

  InferSent (Conneau et al., 2017): max pooling on the hidden states of a biLSTM trained on the Stanford Natural Language Inference dataset

• •

  QuickThoughts (Logeswaran and Lee, 2018): A non-generative version of SkipThought – rather than trying to reconstruct neighbouring sentences, it tries to train a classifier for predicting neighbouring/not neighbouring.

Universal Sentence Encoder (USE) (Cer et al., 2018) is a model developed by Google. The goal of USE is to provide easy-to-use sentence embeddings with good transfer performance.

There are actually 2 implementations of the encoder:

• •

  A Transformer-based model using (self) attention that mean-pools the word embeddings to produce sentence embeddings

• •

  A Deep Averaging Network (DAN) that simply averages word embeddings and bi-gram embeddings and feeds the resulting representation into a straightforward deep neural network. The DAN trades accuracy for performance, with respect to the transformer, which scales with the square of the sentence length.

The models are trained on a set of self-supervised tasks and on SNLI

The unsupervised tasks are trained on Wikipedia, web news, web question-answer pages and discussion forums

One interesting note in the paper mentions that for STS tasks the authors use the angular distance (the arccos of the cosine similarity) as a distance measure, because “arccos converts cosine similarity into an angular distance that obeys the triangle inequality. We find that angular distance performs better on STS than cosine similarity.”

Sentence-BERT (SBERT) (Reimers and Gurevych, 2019) is a BERT based model that uses a Siamese network approach to fine-tune BERT on a set of tasks in order to produce useful sentence embeddings.

For SBERT the goal is to produce embeddings that are primarily useful for computationally and/or combinatorially expensive tasks like clustering and nearest neighbour search – producing vectors that can be used in optimized computation libraries such as numpy or sklearn.

SBERT promises semantically meaningful sentence embeddings (similar sentences are close in vector space).

It tries to accomplish this by pooling the token embeddings while training on STS tasks but using fixed weights between 2 BERTs, so that the resulting embeddings for the two input sentences are created using identical weights. It claims to be better on STS or SentEval than InferSent or USE.

SBERT trains using 3 objectives:

• •

  Classification, by pooling the embeddings of $u$ and $v$ as $[u;v;u-v]$ as input to a softmax, using cross entropy loss

• •

  Regression (presumably on STS), by using cosim(u,v) using MSE as the loss

• •

  Triplet loss: Given 3 sentences; a (anchor), p (positive example) and n (negative example), train the model such that $|a-p|<|a-n|$, or in other words: minimize $|a-p|-|a-n|$.

SBERT is trained on SNLI and MultiNLI. It does well on STS tasks (as expected)

LASER (Artetxe and Schwenk, 2019) is a bit different from the other models in this summary in that it is primarily interested in universal language agnostic sentence embeddings. It is trained on a large dataset consisting of 93 languages. It uses a relative simple biLSTM to encode the input text – BPE tokenized text of any language is fed into the same encoder. This creates a fixed size language agnostic embedding that is then used in the decoder to translate the output into a specified language. The paper claims that fixed length representations are more versatile and compatible than variable length representations: “For instance, there is not always a one-to-one correspondence among words in different languages (e.g. a single word of a morphologically complex language might correspond to several words of a morphologically simple language), so having a separate vector for each word might not transfer as well across languages.”

• •

  The embeddings are results of max pooling the hidden states of layers of the biLSTM of dimensionality 512, concatenating forward and backward hidden state into sentence representations of dimensionality 1024

• •

  The authors claim that the model does well on

  • –

    XNLI (entailment)

  • –

    MLDoc (document classification)

  • –

    BUCC (finding the same sentence in another language)

  • –

    Tatoeba: a new cross language similarity search benchmark

• •

  In the future, the authors would like to improve the result by using a self attention encoder, pre trained word embeddings, and back translation.

DeCLUTR (Giorgi et al., 2020) aims to produce useful universal sentence embeddings by unsupervised learning only. It points out that the best results so far have been obtained by methods that used at least some supervised learning, but that it is important to close the gap between supervised and unsupervised methods for languages and domains for which no supervised data exists.

• •

  Wants to be good at wide variety of tasks

• •

  Most universal sentence embedders train supervised on SNLI or multiNLI (entailment, contradiction, neutral). Examples are InferSent, USE, and SBERT.

• •

  The authors describe Skip-thought as an unsupervised generative model that uses sentence embeddings to predict words in neighbouring sentences. They mention that the generative nature of the model makes it expensive and surface focused. QuickThoughts tries to improve on this by classifying context sentences from non-context sentences rather than generating them.

• •

  The authors describe DeCLUTR’s approach as similar to SBERT, but self-supervised, and the objective as similar to QuickThoughts, but using segments rather than whole sentences.

• •

  Like SBERT’s triplet approach, it uses contrastive loss: It tries to minimize $|a-p|-|a-n|$.

• •

  Like QuickThoughts, it classifies sentences (or rather sentence segments) as near by or far away.

This is simply the TfidfVectorizer model from scikit-learn (https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html). It is a WordVectorizer (bag of words) with a TF-IDF weighting applied to it. It has been included in the experiments as a baseline for the encoder models.

We test the main hypothesis by using a simple model:

• •

  For every dataset, we cluster the data into as many clusters as there are classes. We use k-means clustering.

• •

  For every cluster, we allocate the most frequently occurring class within the cluster as its designated class

• •

  This is the model. Prediction is simply a mapping from the predicted cluster of an example to the designated class of that cluster

• •

  F1 (macro averaged) will give a measure of how well the model performs

For establishing a baseline on the classifier models, we use logistic regression on the embedding data to train a linear model.

If we count the number of tasks on SentEval for which each model has the highest score (see Table 2, we end up with the following list:

1. 1.

   DeCLUTR (10 wins)

2. 2.

   SBERT (8 wins)

3. 3.

   LASER (6 wins)

4. 4.

   USE (2 wins)