Jointly Learning Truth-Conditional Denotations and Groundings using Parallel Attention

The meaning of words and phrases is very flexible. An expression like wooden table can refer to an enormous variety of styles of table, kinds of wood, etc. and can be used across a large variety of different conversational contexts and situations. To capture this flexibility, many theories of semantics break down word meaning into two parts. First, word denotations capture situation-independent aspects of meaning. The denotation of table, for instance, might encode features that are associated with tablehood—like having a broad flat surface. In truth conditional semantics, denotations are often modeled using boolean-valued predicates like table($\cdot$) (see, e.g., Heim and Kratzer, 2000; McConnell-Ginet and Chierchia, 2000). Second, groundings associate words with information in some specific situation (Harnad, 1990). A grounding for table could indicate, amongst other things, what region of space in a room is occupied by the table in question. Groundings are often modeled by the binding of situation-specific information to grounding variables which are passed to denotations, for example, $x$ in table($x$).

Denotation and grounding are tightly linked. For instance, while a region of space, a visual texture, or a particular configuration of shapes are appropriate kinds of information for evaluating tablehood, different information is appropriate for evaluating the meaning of a spatial relation such as on top. Groundings are flexible and influenced by language. For example, the denotations of individual words can require grounding in collections of objects (e.g., a forest). Groundings can also be influenced by social context. The objects depicted in a painted mural are relevant during a lecture on art history, but will be ignored when discussing how to rearrange furniture in a room.

Recent years have seen a great deal of work in language grounding in domains such as images and video; however, most of this work does not make use of the truth-conditional framework with its advantages in modularity and compositionality (see §2). At the same time, little work from within linguistic semantics has addressed the problem of learning the complex relationships between denotations and groundings just mentioned.

In this paper, we address this gap, studying the problem of simultaneously learning the meaning of words and their groundings in a truth-conditional setting. Building on the work of Mao et al. (2019), we show how to use a novel parallel attention mechanism to bind perceptual information to grounding variables representing visual objects in the CLEVR dataset (Johnson et al., 2017a).

Our system achieves state-of-the-art performance on the visual question answering task (VQA). Critically, unlike Mao et al. (2019) our model is trained end-to-end and makes no use of an external object-recognition module. Most importantly, the model is able to learn how to bind grounding variables to percepts by back-propagating entirely through the compositional semantics, with no other external training signal. We perform a number of ablation experiments to better understand which components of our model are critical. Finally, we demonstrate that our model can capture the top-down influence of semantics; we show that the model can acquire non-canonical groundings just by adjusting its linguistic input.

Formal semantics formalizes word meanings using predicates of variables that encode situational groundings (Lakoff, 1970; Montague, 1970). In Montague semantics, these predicates capture truth conditions on word meaning (Heim and Kratzer, 2000; McConnell-Ginet and Chierchia, 2000; Gamut, 1991). However, the use of word-meaning predicates (or functions) is common across many other varieties of semantics (e.g., Pietroski, 2005; Jackendoff, 1990). The advantages of an architecture which distinguishes denotations and groundings in this way is that (i) it provides a clean division of labor between situation-specific (grounding) and situation-general (denotation) aspects of the problem and (ii) it allows for compositionality, in which the meaning of composed expressions can be built up from logical combinations of functions of grounding variables. For example the meaning of wooden table can be composed as wooden($x$) $\wedge$ table($x$).

Linguistic semantics has not historically focused on the problem of how grounding variables get bound to information from outside the linguistic system. By contrast, the last years have seen an explosion of work that tackles this problem ofgrounded language learning. This includes linguistically inspired grounding models from computational linguistics and AI (Matuszek et al., 2012; Krishnamurthy and Kollar, 2013; Yu et al., 2015; Siskind, 1992) or robotics (Kollar et al., 2013; Tellex et al., 2014), computer vision models that answer questions about images, (Antol et al., 2015; Geman et al., 2015) or videos (Tapaswi et al., 2016), and reinforcement learning models of how grounded meaning can be learned by trial and error in a simulated 3D world (Hermann et al., 2017).

Early work in grounded language learning made use of predicate-based representations of meaning similar to those used in linguistics (e.g., Siskind, 1992), and probabilistic variants of these approaches continue to be explored in AI (e.g., Matuszek et al., 2012; Yu et al., 2015; Krishnamurthy and Kollar, 2013) and robotics (e.g., Tellex et al., 2014; Kollar et al., 2013). However, these approaches are typically non-differentiable and, for this reason, most current work in the area makes use of jointly trained neural models (Perez et al., 2017; Hudson and Manning, 2018; Anderson et al., 2018). Although some of these models do feature dedicated word-specific neural functions (that is, modules, see e.g., Andreas et al., 2016; Johnson et al., 2017b), these functions are not predicates in the usual sense. Instead, they usually return distributed or attention-based representations of word meaning which can limit their ability to generalize systematically (Bahdanau et al., 2019).

One recent strand of work—the neurosymbolic approach of Mao et al. (2019)—attempts to combine the truth-conditional and neural approaches to grounded language learning. By making use of a differentiable variant of truth-conditional semantics—roughly, a fuzzy-logic-based semantics—Mao et al. (2019) show how meanings can be learned in a predicate-based but fully differentiable system. Mao et al. (2019) are able to achieve near-perfect performance on the CLEVR visual question answering dataset.

There is one important limitation of the work of Mao et al. (2019). While their model learns denotations of individual words, it does not jointly learn groundings for those words. Instead, the model makes use of an out-of-the-box, pretrained object recognizer to find bounding boxes around candidate objects in images. These bounding boxes are then used to derive the featural representations of objects which are bound to grounding variables in the model. Because these bounding boxes are provided by an external system, they cannot be influenced by the linguistic system.

In this work, we extend the model of Mao et al. (2019) to jointly learn denotations and groundings through a novel mechanism which we call parallel attention. The task is closely related to recent work on self-supervised learning of object representations (Burgess et al., 2019; Engelcke et al., 2019; Greff et al., 2019, 2016; Jiang et al., 2020; Lin et al., 2020; Locatello et al., 2020). That work aims to learn object representations without bounding box supervision, though the training signal differs from the current work, using reconstruction loss instead of linguistic signal. We describe our model in the next section.

Our model consists of two components: a grounding module and a semantic module. The grounding model processes an input image using our novel parallel attention mechanism to produce a set of $n$ grounding variables $\{v_{i}\}_{1\leq i\leq n}$ each of which is meant to represent an object, group of objects, or other groundable part of a scene. In order to allow for differing numbers of objects across images, the grounding module also returns a vector $h\in[0,1]^{n}$ of objecthood parameters which allows the model to turn off or on particular grounding variables—allowing our model to learn the number of objects in a scene from data.

The semantic module represents individual word denotations as well as how these are composed. The meaning of CLEVR questions in our model is represented using a domain specific language (DSL) derived from that of Mao et al. (2019) and related to the DSL reported in the original CLEVR paper (Johnson et al., 2017a).

At a high level, our semantics, like that of Mao et al. (2019), makes use of two leading ideas. First, the denotations of individual predicates used in CLEVR questions, such as red or in front of, are formalized in terms of concept embeddings which are compared to grounding variables to determine the degree to which an object matches the predicate. Second, composition is accomplished primarily through fuzzy variants of standard logical operators such as and and or. We describe the model in greater detail below.

In order to better understand the behavior of our model, we performed several experiments on variant models which modify or ablate features of the architecture. We describe these in this section.

Models were optimized using the AdaBelief optimizer (Zhuang et al., 2020), with a learning rate of $4\cdot 10^{-5}$ and batch size of 24. Each model was trained on a single RTX 2080 Ti. Following Mao et al. (2019), we used a curriculum learning strategy which incrementally introduced scenes and questions in the order of their complexity; scene complexity was measured by number of objects and question complexity by length of semantic parse and whether they included two-place spatial relations. All simulations assume access to ground truth semantic parses.

We first evaluate the model on the CLEVR dataset, a popular synthetic benchmark for question answering based on visual scenes of 3D-rendered objects (Johnson et al., 2017a). The CLEVR training set consists of 70,000 images and 700,000 question/answer pairs, together with a validation set of 15,000 images and 150,000 question/answer pairs.

We evaluated the performance of our model on CLEVR in two ways. First, we evaluate accuracy on the question-answering task. Second, we evaluate whether the model is able to recover the correct attributes and spatial relations for objects in the scene—that is, whether the model can recover the correct scene graph of the image.

Table 1 shows model accuracy on the CLEVR validation set for our model, the ablated/variants models discussed in §4, and a number of state of the art models from the literature. As can be seen in the table, our base model (Parallel Attention) achieves state-of-the-art performance on CLEVR question answering. Interestingly, the best performance is exhibited by the sequential variant of our model, which we discuss below.

Table 2 shows the results of the scene-graph evaluations on the CLEVR dataset. We report the model’s ability correctly recover the values of particular attributes of individual objects (i.e., red or cube). Recall measures the percentage of correctly identified attribute values over the total number of gold standard object attribute values, and precision measures the percentage of correctly identified attribute values over the total number of predicted object attribute values. Similarly, we report precision and recall for spatial relations (i.e., behind or left).

As can be seen from Tables 1 and 2, eliminating the foreground initialization leads to significant degradations in performance in both question answering and scene graph recovery; eliminating attention scopes leads to a smaller degradation. It appears to be important to allow each grounding variable attention to start from a specific local maximum in the foreground map and to adjust based on what the other grounding variables are looking at. Interestingly, this degradation in performance on scene graphs is much higher in terms of recall suggesting that without foreground initialization, the model is finding fewer of the correct objects.

Our base model assumes that all attentions are updated completely in parallel and symmetrically; symmetry between grounding variables is only broken by the foreground initialization. The sequential and recurrent variants of the model relax this assumption and introduce an ordering into the parallel attention mechanism. Interestingly, the sequential model seems to improve the overall performance of the model, and gives what we believe is the current state of the art for CLEVR question answering. We leave exploring this architecture variant more thoroughly to future work. By contrast, the recurrent architecture leads to a substantial decrease in performance. We note however that this architecture exhibited unstable training performance, suggesting the recurrent connections led to gradient explosions.

As we discussed in the introduction, one of the motivations behind this work is to explore how it is possible to learn the tight contingencies between the way that language is used and the kinds of groundings that are available in a particular situation. To explore this question, we perform several experiments to illustrate the top-down influence of linguistic usage on the kinds of grounding that our model learns. In each of these experiments, we train the system to acquire some non-canonical grounding that goes beyond the simple object groundings of the original CLEVR dataset. These experiments make use of the same set of scenes and questions as the standard CLEVR dataset—only the answers to questions are modified. Thus, the linguistic input provides the only training signal for the non-canonical groundings.

In this paper, we have introduced a model which jointly learns the denotation of words and how to ground them in images. The model extends the architecture of (Mao et al., 2019) with a novel parallel attention mechanism which allows it to combine the advantages of truth-conditional semantics and attention-based grounding in a differentiable system. We achieve state of the art performance on question answering on the CLEVR dataset. We also show that model is able to learn to ground the meaning of linguistic expressions in novel and non-canonical ways based purely on changes to the linguistic input. We hope that the model can serve as an early step in studying the intricate problem of learning the contingencies between linguistic meaning and grounding in a way that is able to take advantage of tools from both machine learning and formal semantics.