Learning Action-Effect Dynamics from Pairs of Scene-graphs

The task aims at understanding changes caused over an image by performing an action described in natural language and then answering a reasoning question over the resulting scene. Figure 2 shows an example from the dataset;

• •

  Inputs:

  1. 1.

     Image (I)- Visual scene with rendered objects

  2. 2.

     Action Text (T_A)- Text describing action to be performed over I

  3. 3.

     Question (Q_H)- Question to assess the system’s ability to understand changes caused by T_A on I

• •

  Output: Answer (A) for the given Q_H

• •

  Answer Vocabulary: [0-9, yes, no, cylinder, sphere, cube, small, big, metal, rubber, red, green, gray, blue, brown, yellow, purple, cyan]

• •

  Evaluation: 27-way Answer Classification / Accuracy (%)

The CLEVR_HYP dataset assumes to have a closed set of object attributes, action types, and question reasoning types.

• •

  Object attributes: 5 colors, 3 shapes, 2 sizes, 2 materials

• •

  Action types: Add object, Remove object(s), Change attribute, Move object (in-plane and out-of-plane)

• •

  Reasoning types: Count objects, Compare objects, Existence of objects, Query attribute and Compare attribute

The dataset is divided into the following partitions;

• •

  Train (67.5k) / Val (13.5k) sets have $<$I, T_A, Q_H, A$>$ tuples along with the scene-graphs as a visual oracle and functional programs¹¹1Originally introduced in CLEVR (Johnson et al. 2017), ex. question ‘How many red metal things are there?’ $\sim$ functional program ‘count(filter_color(filter_material(scene(),metal),red))’ as a textual oracle.

• •

  Test sets consist of only $<$I, T_A, Q_H, A$>$ tuples, and no oracle annotations available. There are three test sets,

  1. 1.

     Ordinary test (13.5k) consists of examples with the same difficulty as train/val

  2. 2.

     2HopT_A test (1.5k) consists of examples where two actions are performed ex. ‘Move a purple object on a red cube then paint it cyan.’

  3. 3.

     2HopQ_H test (1.5k) consists of examples where two reasoning types are combined ex. ‘How many objects are either red or cylinder?’

Following are two top-performing baselines reported in Sampat et al. (2021), to which we will compare the results of our proposed approach in this paper.

• •

  (TIE) Text-conditioned Image Editing: Text-adaptive encoder-decoder with residual gating (Vo et al. 2019) is used to generate new image conditioned on the action text. Then, new image along with the question is fed into LXMERT (Tan and Bansal 2019) (which is a pre-trained vision-language transformer), to generate an answer.

• •

  (SGU) Scene-graph Update: In this model, understanding changes caused by an action text is considered as a graph-editing problem. First, an image is converted into a scene-graph and action text is converted into a functional program (FP). Sampat et al. (2021) developed a module inspired by Chen et al. (2020) that can generate an updated scene graph provided the initial scene-graph and a functional program of action text. It is followed by a neural-symbolic VQA model (Yi et al. 2018) that can generate an answer to the question provided the updated scene-graph.

This stage comprises an ‘Action Encoder’ and an ‘Effect Decoder’. As described in previous section, the training set of CLEVR_HYP provides oracle annotations for an initial scene-graph S (using which the image is rendered) and a scene-graph after executing the action text S^′. We take a random subset of 20k scene-graph pairs from the training set balanced by action types (add, remove, change, move) to train this encoder-decoder.

We capture the difference between states S and S^′ i.e. A${}_{S,S^{\prime}}$ using the encoder. At the test time, we do not have the updated scene-graph S^′ available. To address this issue, the encoder is followed by a decoder, which can reconstruct S^′ provided S and the learned scene difference A${}_{S,S^{\prime}}$. We jointly train encoder-decoder with the following objective;

$$argmax_{\Theta_{ActionEncoder}\Theta_{EffectDecoder}}\\ [logP({\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}S^{\prime}}|\frac{}{}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}S},\frac{}{}ActionEncoder({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}S},{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}S^{\prime}}))]$$

(1)

We assume changes in the scene as a function of action. However we can obtain A_rep i.e. vector representation of natural language action, and it can be trained to approximate A${}_{S,S^{\prime}}$ (from Stage-1). Specifically, we freeze encoder-decoder trained in Stage-1 and learn ‘Natural language to Action Representation’ module that maximizes the following log probability;

$$argmax_{\Theta_{NL2ActionRep}}\\ [logP({\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}S^{\prime}}|\frac{}{}{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}S},\frac{}{}NL2ActionRep({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}T_{A}}))]$$

(2)

Inside this module lies LSTM encoder, which precedes by an embedding layer and followed by dense layers. During training, in addition to finding the values for the weights of the LSTM and dense layers, the word embeddings for each word in the training set are computed. This way, a fixed length vector is generated for each word in the vocabulary depending on the position of the word in context and updated using back-propagation. The LSTM has a hidden layer of size 200.

Stage-3 combines modules trained in Stage-1 and Stage-2 with off-the-shelf ‘Image to Scene-graph’ and ‘Scene-graph Question Answering’ networks. Specifically, ‘Image to Scene-graph’ is Mask R-CNN (He et al. 2017) followed by a ResNet-34 (He et al. 2016), that classifies visual attributes- color, material, size, and shape and obtains 3D coordinates of each object in the scene. The ‘Scene-graph Question Answering’ network is based on (Yi et al. 2018), which has near-perfect accuracy on the scene-graph question answering task over CLEVR (Johnson et al. 2017).

In this section, we discuss performance of our model quantitatively and qualitatively. We also discuss three ablations conducted for our model.

Our experimental results are summarized in Table 1. Our proposed approach (ARL) outperforms existing baselines by 5.9%, 4.8% and 4.2% on Ordinary, 2HopT_A Test and 2HopQ_H Test respectively. This demonstrates that our model not only achieves better overall accuracy but also has improved generalization capability when multiple actions have to be performed on the image or understand logical combinations of attributes while performing reasoning.

In Figure 4, we visually demonstrate scene-graphs predicted by our ARL model over a variety of action texts. From examples 1-2, we can observe that the model can correctly identify objects that match the object attributes (color, size, shape, material) provided in the action text. Examples 3-4 demonstrate that our system is consistent in predictions when we use synonyms of various words (e.g. sphere$\sim$ball, shiny$\sim$metallic) in the dataset. Finally, examples 5-6 show that our model does reasonably well on other actions (add and change).

We further generate a t-SNE plot of action vectors learned by our best proposed model, which is shown in Figure 5. At a first glance, we can say that the learned action representations formulate well-defined and separable clusters corresponding to each action type. Clusters for add, remove and change actions are closer and somewhat overlapping.