Measuring Spurious Correlation in Classification:
”Clever Hans” in Translationese

The goal is to investigate the amount and distribution of topic signal in $O$ and $T$ data, that could be used as a spurious signal in translationese (i.e. $O$ and $T$) classification. As initially we do not know anything about possible topics and their distribution in the data, we use standard approaches to unsupervised topic modeling, like Latent Dirichlet Allocation (LDA) Blei et al. (2001) and BERTopic Grootendorst (2022). Both LDA and BERTopic will cluster our data into classes, i.e. topics. How can we measure whether the topics established by the topic model are potentially relevant to translationese classification? Topics are relevant to $O$ and $T$ translationese classification if the paragraphs in each of the topics are either mostly $O$ paragraphs or if they are mostly $T$ paragraphs, in other words if topics are well aligned to either $O$ or $T$. If this is the case, a translationese classifier may learn to use topic, rather than proper translationese signals (or a mix of both) in translationese classification. To give a simple (and extreme) example, suppose we take the union²²2As $O$ and $T$ paragraphs are disjoint, this is the same as their concatenation. of $O$ and $T$ and cluster the union using LDA into, say, two topics (classes) top₁ and top₂. If (and this is the extreme case) top₁ = $O$ and top₂ = $T$ (or vice versa, i.e. top₁ = $T$ and top₂ = $O$), then topic perfectly predicts $O$ and $T$. We would like our measure to capture this, and we would like the measure to be symmetric, i.e. give the same result no matter whether top₁ = $O$ and top₂ = $T$ or vice-versa. Now consider another (extreme) case: lets say top₁ is half $O$ and half $T$, with top₂ the same but with the other halfs of $O$ and $T$. In this case topic is not able to distinguish between $O$ and $T$ (beyond chance). What about cases in between the two extreme cases? Lets say, top₁ is 3/4 $O$ and 1/4 $T$, and therefore top₂ is 1/4 $O$ and 3/4 $T$ (or vice versa - swap top₁ and top₂). In this case topics top₁ and top₂ are pretty good indicators of $O$ and $T$, and a translationese classifier may pick up on topic signals rather than just translationese proper.

To design a measure that captures the relevance of topic classes to (binary) translationese classification, we need our measure to be symmetric, generalize to more than 2 topic classes, and factor in possible $O$ and $T$ class imbalance. To keep things simple³³3We could design an entropy-based measure of the distribution of topic classes with respect to $O$ and $T$, factor in classification probabilities of LDA etc., here we present a straightforward and easy to use measure we call alignment of topic top_i with $O$ and $T$, denoted

It is easy to see that the definition generalizes to $n$ topic classes top₁ to top_n, that it adapts to different top_i topic sizes as well as the class imbalance between $O$ and $T$⁴⁴4To see this, note that as $O$, $T$ partition the data and as $\bigcup\textrm{\emph{top}}_{i}$ also partition the data, if, let us say, $O\ll T$ and for some top_i = $O$, then there is nothing of $O$ left to any of the other top_j≠i. Note that in our data we have $|O|\approx|T|$).. align_O,T(top_i) $\in$ [0.5, 1] where align_O,T(top_i) = 1 signals perfect alignment of topic top_i with one of $O$ or $T$, and that align_O,T(top_i) = 0.5 signals that top_i is maximally undecided with respect to $O$ and $T$. And the same for avg_align_O,T(top).

Our alignment-based measure⁵⁵5One of our reviewers suggested we compare our measure to existing cluster quality measures. is in fact the same as cluster purity Zhao and Karypis (2001) defined as

We use LDA Blei et al. (2001) as our main unsupervised topic model, as it provides a standard and well-understood baseline⁶⁶6We use the Gensim Rehurek and Sojka (2011) implementation of Mallet LDA McCallum (2002).. LDA makes two key assumptions: (1) documents are a mixture of topics, and (2) topics are a mixture of words. LDA generates a document-term matrix (DTM), where each document is represented by a row and the terms (words) corresponding to each document are represented by the columns. The DTM is decomposed into a document-topic matrix and a topic-word matrix. LDA assigns every word to a latent topic (top_i) through iteration, computing a topic word distribution ($\theta$) in the data. To build this distribution, LDA uses two parameters: $\alpha$ which controls the per-document topic distribution, and $\beta$ (the Dirichlet parameter) which controls the per-topic word distribution. LDA requires us to specify the number of topics $n$ in advance. In our experiments we explore $n$ over three orders of magnitude, roughly doubling $n$ at each step, starting with $n=2$ and going up to $n=500$.

As LDA requires us to specify $n$ in advance, we also use BERTopic Grootendorst (2022), which can find an optimal $n$ given the data⁷⁷7We partly do this as a sanity check to assess whether our steps increasing $n$ across three orders of magnitude for LDA missed an important region of number of topics.. By default, BERTopic utilizes contextual sentence embeddings (SBERT), dimensionality reduction (UMAP), clustering (HDBSCAN), tokenizing (CountVectorizer), and a weighing scheme (c-TFIDF) to perform topic modeling. We choose the embedding model ’T-Systems-onsite/cross-en-de-roberta-sentence-transformer’ from Huggingface Wolf et al. (2020) and the defaults for all other modules.

For LDA, we explore a number of topics $n$ with $n=2,5,10,20,30,50,100,200,300,400$ and $500$, over three orders of magnitude. BERTopic returns 207 topics⁸⁸8BERTopic is stochastic due to UMAP and returns a different number of topics for each run, however, the differences are small..

For each document (here paragraph), we use the highest probability LDA or BERTopic topic assigned to the document to label the document. A topic is then represented by the set of documents labeled with the topic. For each topic top_i, we compute how well the topic is aligned with $O$ and $T$, i.e. we compute align_O,T(top_i), and the weighted average over the topics: avg_align_O,T(top₁, …, top_n).

We plot avg_align_O,T(top₁, …, top_n) in Fig. 1, varying $n$ from $2$ to $500$, at each step roughly doubling $n$ for LDA, and with $n=207$ for and as determined by BERTopic. An average alignment of 0.5 (the dashed green line) shows topics maximally undecided with respect to $O$ and $T$, while a score of 1 indicates perfect alignment where topics completely predict $O$ and $T$. Fig. 1 shows topic alignment with $O$ and $T$ in the range of 0.55 to 0.62, depending on $n$, with BERTopic achieving the overall highest score of 0.62 at $n=207$. For LDA, scores are highest (0.611 - 0.618) for $n=10,20$, and $30$. For good choices of topic numbers $n$, both LDA and BERTopic topics are able to predict $O$ and $T$ (i.e. translationese) by close to 0.62.

This is an interesting and perhaps somewhat surprising result, but what exactly does it mean? There are two important caveats:

First, the fact that topic is able to predict $O$ and $T$ by close to 0.62 in the data does not necessarily mean (i.e. prove) that a high-performance BERT translationese classifier, such as the one presented in Pylypenko et al. (2021), necessarily uses spurious topic information aligned with $O$ and $T$ in the data. At the same time, however, it cannot be ruled out. As a sanity check we tested how well a BERT classifier can learn to predict LDA topic classes for $n=2,10,20$ and $30$ and for BERTopic’s $207$. The results are $0.83,0.64,0.42,0.44$ and $0.57$ (all acc. and well above the largest class baseline). Given how well BERT-based classifiers can pick up patterns in the data, it is prudent to assume that BERT will be sensitive to and use topic signals spuriously aligned with $O$ and $T$.

Second, we cannot at this stage completely rule out (other than perhaps through laborious manual inspection) that some LDA or BERTopic topics may in fact reflect genuine rather than spurious signals. LDA, e.g., uses lexical information, and perhaps some such information is a genuine translationese signal (unlike the place names clearly identified as a spurious topic signal in Amponsah-Kaakyire et al. (2022)), such as e.g. certain forms of verbs (see Section 5 on the Known Unknowns).

The two caveats are aspects of the ”unknown unknowns” we are chasing in this part of the paper. We have a clear indication that topic aligns with and hence can predict $O$ and $T$ in our data up to 0.62. There are very good reasons (but no proof) to assume that it is likely that a high-performance BERT classifier may use this, while we cannot completely rule out that some of the supposedly spurious topic signal may actually be genuine translationese. Given this, 0.62 is an upper bound of how well topic may predict $O$ and $T$ in our data. We may be well advised to take inspiration from the concept of a ”noise floor” in Electronic Engineering. The noise floor is the hum and hiss (of a circuit) due to the components when there is no signal, and below which we cannot identify a signal. Given our findings, perhaps we should regard the 0.62 topic alignment with $O$ and $T$ in our data as a ”topic floor” for translationese classification. This is in fact the recommendation we take from our work: instead of using 0.5 as a random baseline for our (roughly) balanced binary translationese data set, we should require 0.62 as established by the topic alignment experiments as a safe(r) baseline. Put differently, given our data we cannot really be sure about $O$ and $T$ classification results $\leq 0.62$. Suffice it to say, even with a 0.62 baseline, (most of) the classifiers presented in Pylypenko et al. (2021); Amponsah-Kaakyire et al. (2022) easily surpass that baseline (with acc $\geq 0.9$).

We focus on a scenario where we identify (and later mask) NEs automatically, rather than manually. While automatic NER is noisy, unlike potentially high-quality manual NER, it scales and constitutes a realistic application scenario. To assess NER performance, we experiment with a number of SOTA NER models, namely SpaCy Honnibal and Montani (2017), FLERT Schweter and Akbik (2020), multilingual-BERT Devlin et al. (2019b), XLM-RoBERTa Conneau et al. (2020), DistilBERT Sanh et al. (2019), and BERT-German Chan et al. (2020), comparing F1, precision and recall for each of the models against a gold standard NE tagged dataset Agerri et al. (2018). The gold standard consists of 800 sentences from the Europarl German data manually annotated following the ConLL 2002 Tjong Kim Sang (2002) and 2003 Tjong Kim Sang and De Meulder (2003) guidelines, with a total of 433 named entities.

Table  1 shows that BERT-German (a fine-tuned version of bert-base-multilingual-cased on the German WikiANN dataset) has the highest precision, second-highest recall and the highest F1 (0.52) among all the NER models. Hence, we choose BERT-German for all our NER experiments.

To quantify the impact of NE-based spurious topic information on translationese classification, we modify our data by masking NEs (Section 5.2.1), and, in a separate experiment, we explore full masking of the data using POS (Section 5.2.2).

Following Pylypenko et al. (2021), we use multilingual BERT Devlin et al. (2019b) (BERT-base-multilingual-uncased) and fine-tune BERT on the training set of our data set using the Huggingface library. We use a batch size of 16, a learning rate of $4\cdot 10^{-5}$, and the Adam optimizer with $\epsilon=1\cdot 10^{-8}$.

We compare our models with the BERT model reproduced from Pylypenko et al. (2021): a pre-trained BERT-base model (12 layers, 768 hidden dimensions, 12 attention heads) fine-tuned on translationese classification using unmasked data.

Amponsah-Kaakyire et al. (2022) use IG attribution scores to show that BERT utilizes spurious correlations in the data, for example, German $T$ data translated from Spanish contain mentions of Spanish geographical areas, such as ’Spanien’, ’Barcelona’ etc. as top tokens identified by IG. Here we use IG on BERT trained on masked data, and compute the top tokens with the highest attribution scores on average across the masked test sets. We also compute the top POS tags by performing IG on BERT trained on fully POS-tagged data.