Predicting emergent linguistic compositions through time:
Syntactic frame extension via multimodal chaining

The problem of syntactic frame extension concerns the cognitive theory of chaining Lakoff (1987); Malt et al. (1999). It has been proposed that the historical growth of linguistic categories depends on a process of chaining, whereby novel items link to existing referents of a word that are close in semantic space, resulting in chain-like structures. Recent studies have formulated chaining as models of categorization from classic work in cognitive science. Specifically, it has been shown that chaining may be formalized as an exemplar-based mechanism of categorization emphasizing semantic neighborhood profile Nosofsky (1986), which contrasts with a prototype-based mechanism that emphasizes category centrality Reed (1972); Lakoff (1987); Rosch (1975). This computational approach to chaining has been applied to explain word meaning growth in numeral classifiers Habibi et al. (2020) and adjectives Grewal and Xu (2020). Unlike these previous studies, we consider the open issue whether cognitive mechanisms of chaining might be generalized to verb frame extension which draws on rich sources of knowledge. It remains critically undetermined how “shallow models” such as the exemplar model can function or integrate with deep neural models Mahowald et al. (2020); McClelland (2020), and how it might fair with the alternative mechanism of prototype-based chaining in the context of verb frame extension. We address both of these theoretical issues in a framework that explores these alternative mechanisms of chaining in light of probabilistic deep categorization models.

The recent surge of interest in NLP on diachronic semantics has developed Bayesian models of semantic change (e.g., Frermann and Lapata, 2016), diachronic word embeddings (e.g., Hamilton et al., 2016), and deep contextualized language models (e.g., Rosenfeld and Erk, 2018; Hu et al., 2019; Giulianelli et al., 2020). A common assumption in these studies is that linguistic usages (from historical corpora) are sufficient to capture diachronic word meanings. However, previous work has suggested that text-derived distributed representations tend to miss important aspects of word meaning, including perceptual features Andrews et al. (2009); Baroni and Lenci (2008); Baroni et al. (2010) and relational information Necşulescu et al. (2015). It has also been shown that both relational and perceptual knowledge are essential to construct creative or figurative language use such as metaphor Gibbs Jr. et al. (2004); Gentner and Bowdle (2008) and metonymy Radden and Kövecses (1999). Our work examines the function of multimodal semantic representations in capturing diachronic verb-noun compositions, and the extent to which such representations can be integrated with the cognitive mechanisms of chaining.

Computational research has shown the effectiveness of grounding language learning and distributional semantic models in multimodal knowledge beyond linguistic knowledge Lazaridou et al. (2015); Hermann et al. (2017). For instance, Kiros et al. (2014) proposed a pipeline that combines image-text embedding models with LSTM neural language models. Bruni et al. (2014) identifies discrete “visual words” in images, so that the distributional representation of a word can be extended to encompass its co-occurrence with the visual words of images it is associated with. Gella et al. (2017) also showed how visual and multimodal information help to disambiguate verb meanings. Our framework extends these studies by incorporating the dimension of time into exploring how multimodal knowledge predicts novel language use.

Our framework also builds upon recent work on memory-augmented deep learning Vinyals et al. (2016); Snell et al. (2017). In particular, it has been shown that category representations enriched by deep neural networks can effectively generalize to few-shot predictions with sparse input, hence yielding human-like abilities in classifying visual and textual data Pahde et al. (2020); Singh et al. (2020); Holla et al. (2020). In our work, we consider the scenario of constructing novel compositions as they emerge over time, where sparse linguistic information is available. We therefore extend the existing line of research to investigate how representations learned from naturalistic stimuli (e.g., images) and structured knowledge (e.g., knowledge graphs) can reliably model the emergence of flexible language use that expresses new knowledge and experience.

We denote a predicate verb as $v$ (e.g., drive) and a syntactic relation as $r$ (e.g., direct object of a verb), and consider a finite set of verb-syntactic frame elements $f=(v,r)\in\mathcal{F}$. We define the set of nouns that appeared as arguments for a verb (under historically attested syntactic relations) up to time $t$ as support nouns, denoted by $n_{s}\in S(f)^{(t)}$ (e.g., horse appeared as a support noun—the direct object—for the verb drive prior to 1880s). Given a query noun $n^{*}$ (e.g., car upon its emergence in 1880s) that has never been an argument of $v$ under relation $r$, we define syntactic frame extension as probabilistic inference in two related problems:

• 1.

  Verb syntax prediction. Here we predict which verb-syntactic frames $f$ are appropriate to describe the query noun $n^{*}$, operationalized as $p(f|n^{*})$ yet to be specified.

• 2.

  Noun argument prediction. Here we predict which nouns $n^{*}$ are plausible novel arguments for a given verb-syntax frame $f$, operationalized as $p(n^{*}|f)$ yet to be specified.

We solve these inference problems by modeling the joint probability $p(n^{*},f)$ for a query noun and a candidate verb-syntactic frame incrementally through time as follows:

	$\displaystyle p(n^{*},f)^{(t)}$	$\displaystyle=p(n^{*}|f)^{(t)}p(f)^{(t)}$		(1)
		$\displaystyle=p(n^{*}|S(f)^{(t)})p(f)^{(t)}$		(2)

Here we construct verb meaning based on its existing support nouns $S(f)^{(t)}$ at current time $t$. We infer the most probable verb-syntax usages for describing the query noun (Problem 1) as follows:

	$\displaystyle p(f|n^{*})$	$\displaystyle=\frac{p(n^{*},f)^{(t)}}{\sum_{f\in\mathcal{F}}p(n^{*},f)^{(t)}}$		(3)
		$\displaystyle=\frac{p(n^{*}|S(f)^{(t)})p(f)^{(t)}}{\sum_{f^{\prime}\in\mathcal{F}}p(n^{*}|S(f^{\prime})^{(t)})p(f^{\prime})^{(t)}}$		(4)

In the learning phase, we train our model incrementally at each time period $t$ by minimizing the log joint probability $p(n^{*},f)^{(t)}$ in Equation 1 for every frame $f$ and each of its query noun $n^{*}\in Q(f)^{(t)}$:

$\displaystyle J=-\sum\limits_{f\in\mathcal{F}^{(t)}}\sum\limits_{n^{*}\in Q(f)^{(t)}}\log p(n^{*},f)^{(t)}$

(5)

For each noun $n$, we consider a time-dependent hidden representation $\textbf{h}_{n}^{(t)}\in\mathbb{R}^{M}$ derived from different sources of knowledge (specified in Section 3.2). For the prior probability $p(f)^{(t)}$, we consider a frequency-based approach that computes the proportion for the number of unique noun arguments that a (verb) frame has been paired with and attested in a historical corpus:

$\displaystyle p(f)^{(t)}=\frac{|S(f)^{(t)}|}{\sum_{f^{\prime}\in\mathcal{F}}|S(f^{\prime})^{(t)}|}$

(6)

We formalize $p(n^{*}|f)$ (Problem 2), namely $p(n^{*}|N(f))^{(t)}$, by two classes of deep categorization models motivated by the literature on chaining, categorization, and memory-augmented learning.

Deep prototype model (SFEM-DPM). SFEM-DPM draws inspirations from prototypical network for few-shot learning Snell et al. (2017) and is grounded in the prototype theory of categorization in cognitive psychology Rosch (1975). The model computes a set of hidden representations for every support noun $n_{s}\in S(f)^{(t)}$, and takes the expected vector as a prototype $\textbf{c}_{f}^{(t)}$ to represent $f$ at time $t$:

$\displaystyle\textbf{c}_{f}^{(t)}=\frac{1}{|S(f)^{(t)}|}\sum_{n_{s}\in S(f)^{(t)}}\textbf{h}_{n_{s}}^{(t)}$

(7)

The likelihood of extending $n^{*}$ to $S(f)^{(t)}$ is then defined as a softmax distribution over $l_{2}$ distances $d(\cdot,\cdot)$ to the embedded prototype:

$\displaystyle p(n^{*}|S(f)^{(t)})=\frac{\exp{(-d(\textbf{h}_{n^{*}}^{(t)},\textbf{c}_{f}^{(t)}))}}{\sum_{f^{\prime}}\exp(-d(\textbf{h}_{n^{*}}^{(t)},\textbf{c}_{f}^{(t)}))}$

(8)

Deep exemplar model (SFEM-DEM). In contrast to the prototype model, SFEM-DEM resembles the memory-augmented matching network in deep learning Vinyals et al. (2016), and formalizes the exemplar theory of categorization Nosofsky (1986) and chaining-based category growth Habibi et al. (2020). Unlike DPM, this model depends on the $l_{2}$ distances between $n^{*}$ and every support noun:

(9)

In addition to the probabilistic formulation, SFEM draws on structured knowledge including perceptual, conceptual, and linguistic cues to construct multimodal semantic representations $\textbf{h}_{n}^{(t)}$ introduced in Section 3.1.

Perceptual knowledge. We capture perceptual knowledge from image representations in the large, taxonomically organized ImageNet database Deng et al. (2009). For each noun $n$, we randomly sample a collection of 64 images from the union of all ImageNet synsets that contains $n$, and encode the images through the VGG-19 convolutional neural network Simonyan and Zisserman (2015) by extracting the output vector from the last fully connected layer after all convolutions (see similar procedures also in Pinto Jr. and Xu, 2021). We then average the encoded images to a mean vector $\textbf{x}_{p}(n)\in\mathbb{R}^{1000}$ as the perceptual representation of $n$.

Conceptual knowledge. To capture conceptual knowledge beyond perceptual information (e.g., attributes and functions), we extract information from the ConceptNet knowledge graph Speer et al. (2017), which connects concepts in a network structure via different types of relations as edges. This graph reflects commonsense knowledge of a concept (noun) such as its functional role (e.g., a car IS_USED_FOR transportation), taxonomic information (e.g., a car IS_A vehicle), or attributes (e.g., a car HAS_A wheel). Since the concepts and their relations may change over time, we prepare a diachronic slice of the ConceptNet graph at each time $t$ by removing all words with frequency up to $t$ in a reference historical text corpus (see Section 4 for details) under a threshold $k_{\text{c}}$ which we set to be 10. We then compute embeddings for the remaining concepts following methods recommended in the original study by Speer et al. (2017). In particular, we perform singular value decomposition (SVD) on the positive pointwise mutual information matrix $\textbf{M}_{G}^{(t)}$ of the ConceptNet $G^{(t)}$ truncated at time $t$, and combine the top 300 dimensions (with largest singular values) of the term and context matrix symmetrically into a concept embedding matrix. Each row of the resulting row matrix of SVD will therefore serves as the conceptual embedding $\textbf{x}_{c}(n)^{(t)}\in\mathbb{R}^{300}$ for its corresponding noun.

Linguistic knowledge. For linguistic knowledge, we take the HistWords diachronic word embeddings $\textbf{x}_{l}^{(t)}\in\mathbb{R}^{300}$ pre-trained on the Google N-Grams English corpora to represent linguistic meaning of each noun at decade $t$ Hamilton et al. (2016).

Knowledge integration. To construct a unified representation that incorporates knowledge from different modalities, we take the mean of the unimodal representations described into a joint vector $\textbf{x}_{n}\in\mathbb{R}^{300}$, and then apply an integration function $g:\mathbb{R}^{300}\rightarrow\mathbb{R}^{M}$ parameterized by a feedforward neural network to get the multimodal word representation $h_{n}^{(t)}$.¹¹1For ImageNet embeddings, we apply a linear transformation to project each $\textbf{x}_{p}^{(t)}$ into $\mathbb{R}^{300}$ so that all unimodal representations are 300-d vectors before taking the means. Our framework allows flexible combinations of the three modalities introduced, e.g., a full model would utilizes all three types of knowledge, while a linguistic-only baseline will directly take HistWords embeddings $\textbf{x}_{l}^{(t)}$ as inputs of the integration network.

We implemented the integration network $g(\cdot)$ of SFEM as a three-layer feedforward neural network with an output dimension $M=100$, and keep parameters and embeddings in other modules fixed during learning.³³3See Appendix A for additional implementation details. At each decade, we randomly sample $70\%$ of the query nouns with their associated verb-syntactic pairs as training data, and take the remaining examples for model testing such that there is no overlap in the query nouns between training and testing. We trained models on the negative log-likelihood loss defined in Equation 5 at each decade. To examine how multimodal knowledge contributes to temporal prediction of novel language use, we trained 5 DEM and 5 DPM models using information from different modalities.

We test our models on both verb syntax and noun argument predictive tasks with the goals of assessing 1) the contributions of multimodal knowledge, and 2) the two alternative mechanisms of chaining. We also consider baseline models that do not implement chaining-based mechanisms: a frequency baseline that predicts by count in GSN up to time $t$, and a random guesser. We evaluate model performance via standard receiver operating characteristics (ROC) curves that reveal cumulative precision of models in their top $m$ predictions. We compute the standard area-under-curve (AUC) statistics for the ROC curves to get the mean precision over all values of $m$ from 1 to the candidate set size. Figure 3 summarizes the results over the 150 years. We observe that 1) all multimodal models perform better than their uni-/bi-modal counterparts, and 2) the exemplar-based model performs dominantly better than the prototype-based counterpart, and both outperform the baseline models without chaining. In particular, a tri-modal deep exemplar model that incorporates knowledge from all three modalities achieves the best overall performance. These results provide strong support that verb frame extension depends on multimodal knowledge and an exemplar-based chaining.

To further assess how the models perform in predicting emerging verb extension toward both novel and existing nouns, we report separate mean AUC scores for these predictive cases where query nouns are either completely novel (i.e., zero token frequencies) or established. (i.e., above-zero frequencies) at the time of prediction. Table 2 summarizes these results and shows that model performances are similar under four predictive cases. For prediction with novel query nouns, it is not surprising that linguistic-only models fail due to the unavailability of linguistic mentions. However, for prediction with established query nouns, the superiority of multimodal SFEMs is still prominent suggesting that our framework captures general principles in verb frame extension (and not just for predicting verb extension toward novel nouns).

Table 3 compares sample verb syntax predictions made by the full and linguistic-only DEM models that cover a diverse range of concepts including inventions (e.g., airplane), discoveries (e.g., microorganism), and occupations (e.g. astronaut). We observe that the full model typically constructs reasonable predicate verbs that reflect salient features of the query noun (e.g., cars are vehicles that are drive-able). In contrast, the linguistic-only model often predicts verbs that are either overly generic (e.g., purchase a telephone) or nonsensical.

We provide further analyses and interpret why both multimodality and chaining mechanisms are fundamental to predicting emergent verb compositions.