Probing Neural Language Models for Human Tacit Assumptions

Language models (LMs) assign probabilities to sequences of text. They are trained on large text corpora to predict the probability of a new word based on its surrounding context. Unidirectional models approximate for any text sequence $\boldsymbol{w}=[w_{1},w_{2},\dots w_{N}]$ the factorized left-context probability ${p(w)=\prod^{N}_{i=1}p(w_{i}\mid w_{1}\dots w_{i-1})}$. Recent neural bi-directional language models estimate the probability of an intermediate ‘masked out’ token given both left and right context; this task is colloquially “masked language modelling” (MLM). Training in this way produces a probability model that, given input sequence $\boldsymbol{w}$ and an arbitrary vocabulary word predicts the distribution ${p(w_{i}=v\mid w_{1},\dots w_{i-1},w_{i+1},\dots w_{n})}$. When neural bi-directional LMs trained for MLM are subsequently used as contextual encoders,¹¹1That is, when used to obtain contextualized representations of words and sequences. performance across a wide range of language understanding tasks greatly improves.

We investigate two recent neural LMs: Bi-directional Encoder Representations from Transformers (\procBERT) (?, ?) and Robustly optimized BERT approach (\procRoBERTa) (?, ?). In addition to the MLM objective, \procBERT is trained with an auxiliary objective of next-sentence prediction. \procBERT is trained on a book corpus and English Wikipedia. Using an identical neural architecture, \procRoBERTa is trained for purely MLM (no next-sentence prediction) on a much larger dataset with words masked out of larger input sequences. Performance increases ubiquitously on standard NLU tasks when \procBERT is replaced with \procRoBERTa as an off-the-shelf contexual encoder.

Recent research employs word prediction tests to explore whether contextualized language models capture a range of linguistic phenomena, e.g. syntax (?, ?), pragmatics, semantic roles, and negation (?, ?). These diagnostics have psycholinguistic origins; they draw an analogy between the “fill-in-the-blank” word predictions of a pre-trained language model and distribution of aggregated human responses in cloze tests designed to target specific sentence processing phenomena. Similar tests have been used to evaluate how well these models capture symbolic reasoning (?, ?) and relational facts (?, ?).

Recognizing associations between concepts and their defining properties is key to natural language understanding and plays “a critical role in language both for the conventional meaning of utterances, and in conversational inference” (?, ?). Tacit assumption (TAs) are commonly accepted beliefs about specific entities (Alice has a dog) and stereotypic TAs (STAs) pertain to a generic concept, or a class of entity (people have dogs) (?, ?). While held by individuals, STAs are generally agreed upon and are vital for reflexive reasoning and pragmatics; Alice might tell Bob ‘I have to walk my dog!,’ but she does not need to say “I am a person, and people have dogs, and dogs need to be walked, so I have to walk my dog!” Comprehending STAs allows for generalized recognition of new categorical instances, and facilitates learning new categories (?, ?), as shown in early word learning by children (?, ?). STAs are not explicitly facts.²²2E.g., “countries have presidents” does not apply to all countries. Rather, they are sufficiently probable assumptions to be associated with concepts by a majority of people. A partial inspiration for this work was the observation by ? (?) that the concept attributes most supported by peoples’ search engine query logs (?, ?) were strikingly similar to examples of STAs listed by Prince. That is, there is strong evidence that the beliefs people hold about particular conceptual attributes (e.g. “countries have kings”), are reflected in the aggregation of their most frequent search terms (“what is the name of the king of France?”).

Our goal is to determine whether contextualized language models exposed to large corpora encode associations between concepts and their tacitly assumed properties. We develop probes that specifically test a model’s ability to recognize STAs. Previous works (?, ?, ?, ?) have tested for similar types of stereotypic beliefs; they use supervised training of probing classifiers (?, ?) to identify concept/attribute pairs. In contrast, our word prediction diagnostics find that these associations fall out of semi-supervised LM pretraining. In other words, the neural LM inducts STAs as a byproduct of learning co-occurrence without receiving explicit cues to do so.

We construct prompts for evaluating STAs in LMs by leveraging the CSLB Concept Property Norms (?, ?), a large extension of the McRae study that contains $638$ concepts each linked with roughly $34$ associated properties. The fill-in-the-blank prompts are natural language statements in which the target concept associated with a set of human-provided properties is the missing word . If LMs accurately predict the missing concept, we posit that they encode the given STA set. We iteratively grow prompts by appending conceptual properties into a single compound verb phrase (Figure 1) until the verb phrase contains $10$ properties. Since we test for $266$ concepts, this process creates a total of $2,660$ prompts.⁴⁴4Because LMs are highly sensitive to the ‘a/an’ determiner preceding a masked word e.g. LMs far prefer to complete “A    buzzes,” with “bee,” but prefer e.g. “insect” to complete “An    buzzes.”, a task issue noted by ? (?). We remove examples containing concepts that begin with vowel sounds. A prompt construction that simultaneously accepts words that start with both vowels and consonants is left for future work. ? (?) record production frequencies (PF) enumerating how many people produced each property for a given concept. For each concept, we select and append the properties with the highest PF in decreasing order. Iteratively growing prompts enables a gradient of performance - we observe concept retrieval given few “clue” properties and track improvements as more are given.

We use mean reciprocal rank (MRR), or $1/\text{rank}_{\text{LM}}(\text{target concept})$, a metric more sensitive to fine-grained differences in rank than other common retrieval metrics such as recall. We track the predicted rank of a target concept from relatively low ranks given few ‘clue’ properties to much higher ranks as more properties are appended. MRR above $0.5$ for a test set indicates that a model’s top 1 prediction is correct in a majority of examples. We also report the overall probability the LM assigns to the target concept regardless of rank. This allows us to measure model confidence beyond empirical task performance.

Examples of prompts and corresponding model predictions are shown in Appendix Figure 4. We find that model predictions are nearly always grammatical and semantically sensible. Highly-ranked incorrect answers generally apply to a subset of the conjunction of properties, or are correct at an intermediate iteration but become precluded by subsequently appended properties. ⁶⁶6E.g. tiger and lion are correct for ‘A    has fur, is big, and has claws’ but reveal to be incorrect with the appended ‘lives in woods’ We note that an optimal performance may not be perfect; not all prompts uniquely identify the target concept, even at $10$ properties. ⁷⁷7E.g. the properties of buffalo do not distinguish it from cow. However, models still perform nearly as well as could be expected given the ambiguity.

To measure whether the the type of property affects the ability of LMs to retrieve a concept, we create additional prompts that only contain properties of specific categories as grouped by ? (?): visual perceptual (bears have fur), functional (eat fish), and encyclopaedic (are found in forests).⁸⁸8We omit the categories “other perceptual” (bears growl) and “taxonomic” (bears are animals), as few concepts have more than 2-3 such properties.

Figure 3a shows that \procRoBERTa-L performs interchangeably well given just encyclopedic or functional type properties. \procBERT (not shown) shows a similar overall pattern, but it performs slightly better given encyclopedic properties than functional. Perceptual properties are overall less helpful for models to distinguish concepts than non-perceptual. This may be the product of category specificity; while perceptual properties are produced by humans nearly as frequently as non-perceptual, the average perceptual property is assigned to nearly twice as many CSLB concepts as the average non-perceptual (6 to 3). However, the empirical finding coheres with previous conclusions that models that learn from language alone lack knowledge of perceptual features (?, ?, ?).

When designing the probes, we selected and appended the $10$ properties with the highest production frequencies (PF) in decreasing order. To investigate whether these selection and ordering choices affect a model’s performance in the retrieval task, we compare the top-PF property selection method with an alternative selection criterion using the bottom-PF properties. For both selection methods, we compare the decreasing-PF ordering with a reversed, increasing-PF order. We compare the resulting 4 evaluations against a random baseline that measures performance using a random permutation of a randomly-selected set of properties.⁹⁹9 The random baseline’s performance is averaged over 5 random permutations of 5 random sets for each concept.

Figure 3b compare the differences in performance. Regardless of ordering, the selection of the top (bottom)-PF features improves (reduces) model performance relative to the random baseline. Ordering by decreasing PF improves performance over the opposite direction by up to $0.2$ for earlier sizes of property conjunction, but the two strategies converge in performance for larger sizes. This indicates that the selection and ordering criteria of the properties may matter when adding them to prompts. The properties with lower PF are correspondingly less beneficial for model performance. This suggests that assumptions that are less stereotypic—that is, highly salient to fewer humans—are less well captured by the LMs.

We can consider the listed properties as samples from a fuzzy notion of a human STA distribution conditioned on the concept and relation. These STAs reflect how humans codify their probabilistic beliefs about the world. What a subject writes down about the ‘dog’ concept reflects what that subject believes from their experience to be sufficiently ubiquitous, i.e. extremely probable, for all ‘dog’ instances. The dataset also portrays a distribution over listed STAs. Not all norms are produced by all participants given the same concepts and relation prompts; this reflects how individuals hold different sets of STAs about the same concept. Through either of these lenses, we can speculate that the human subject produces the sample e.g. ‘fur’ from some $p(\text{STA}\mid\text{concept }=\text{{bear}},\ \text{relation}=\textit{has})$. ¹²¹²12This formulation should be taken with a grain of salt; the subject is given all relation phrases at once and has the opportunity to fill out as many (or few) completions as she deems salient, provided that in combination there are at least 5 total properties listed. We can consider our protocol to be sampling from a LM approximation of such a conditional distribution.

Asking language models to list properties via word prediction is inherently limiting, as the models are not primed to specifically produce properties beyond whatever cues we can embed in the context of a sentence. In contrast, human subjects were asked directly “What are the properties of X?” (?, ?). This is a highly semantically constraining question that cannot be directly asked of an off-the-shelf language model.

The phrasing of the question to humans also has implications regarding salience: when describing a dog, humans would rarely, if never, describe a dog as being “larger than a pencil”, even though humans are “capable of verifying” this property (?, ?). Even if they do produce a property as opposed to an alternative lexical completion, it may be unfair to expect language models to replicate how human subjects prefer to list properties that distinguish and are salient to a concept (e.g. ‘goes moo’) as opposed to listing properties that apply to many concepts (e.g. ‘has a heart’). Thus, comparing properties elicited by language models to those elicited by humans is a challenging endeavour. Anticipating this issue, we prepend the phrase ‘Everyone knows that’ to our prompts. They therefore take the form shown in the left column of Table 1. For the sake of comparability, we evaluate the models’ responses against only the human responses that fit the same syntax. We also remove human-produced properties with multiple words following the relation (e.g. ‘is found in forests’) since the contextualized LMs under consideration can only predict a single missing word. Our method produces a set of between 495 and 583 prompts for each of the relations considered.

We use the information retrieval metric mean average precision (mAP) for ranked sequences of predictions in which there are multiple correct answers. We define mAP here given $n$ test examples:

where $P_{i}(j)$ = precision@$j$ and $\Delta r_{i}(j)$ is the change in recall from item $j-1$ to $j$ for example $i$. We report mAP on prediction ranks over a LM’s entire vocabulary ($\text{mAP}_{\proc{vocab}}$), but also over a much smaller vocabulary ($\text{mAP}_{\proc{sens}}$) comprising the set of human completions that fit the given prompt syntax for all concepts in the study. This follows the intuition that responses given for a set of concepts are likely not attributes of the other concepts, and models should be sensitive to this discrepancy. While mAP measures the ability to distinguish the set¹³¹³13Invariant to order of correct answers. of correct responses from incorrect responses, we also evaluate probability assigned among the correct answers by computing average Spearman’s $\rho$ between human production frequency and LM probability.

Results using these metrics are displayed in Table 2. We find that \procRoBERTa-L outperforms the other models by up to double mAP. No model’s rank ordering of correct answers correlates particularly strongly with human production frequencies. When we narrow the models’ vocabulary to include only the property words produced by humans for a given syntax, we find that performance ($\text{mAP}_{\proc{sens}}$) increases ubiquitously.

Models generally provide coherent and grammatically acceptable completions. Most outputs fall under the category of ‘verifiable by humans,’ which as noted by McRae et al. could be listed by humans given sufficient instruction. We observe properties that apply to the concept but are not in the dataset ¹⁴¹⁴14E.g. ‘hamsters are real’ and ‘motorcycles have horsepower’. and properties that apply to senses of a concept that were not considered in the human responses. ¹⁵¹⁵15While human subjects list only properties of the object anchor, LMs also provide properties of a television anchor. We find that some prompts are not sufficently syntactically constraining, and license non-nominative completions. The relation has permits past participle completions (e.g. ‘has arrived’) along with the targeted nominative attributes (‘has wheels’). We also find that models idiosyncratically favor specific words regardless of the concept, which can lead to unacceptable completions.¹⁶¹⁶16\procRoBERTa-B often blindly produces ‘has legs’, the two \procBERT models predict that nearly all concepts are ‘made of wood,’ and all models except \procRoBERTa-L often produce ‘is dangerous.’ We provide examples predictions produced by models in Appendix Figure 5.

We investigate the extent to which our choice of lexical framing impacts model performance by ablating the step in which “everyone knows that” is prepended to the prompt. We find a relatively wide discrepancy in effects; with the lessened left context, models perform on average $.05$ and $.1$ mAP worse on the is and has relations respectively, but perform $.06$ and $.01$ mAP better on is a and has a. Notably, \procRoBERTa-L sees a steep drop in performance on the has relation, losing nearly .3 mAP. We observe that models exhibit highly varying levels of instability given the choice of context. This highlights the difficulty in constructing prompts that effectively target the same type of lexical response from any arbitrary bi-directional LM.