Super-CLEVR: A Virtual Benchmark to
Diagnose Domain Robustness in Visual Reasoning

Visual complexity. A major difference between different VQA datasets is visual complexity. For example, in the CLEVR dataset, objects are simple, atomic shapes while in real-world data, objects are more complex and have hierarchical parts. While hard to quantify, visual complexity is related to various factors, such as object variety, object size, background, texture, lighting, occlusion, view point, etc. In our work, we control visual complexity by introducing more challenging objects that can have distinct attributes associated with their parts, and by optionally pasting various textures onto objects. Examples of generated images with different complexity levels are shown in Fig. 1.

Question redundancy. Question redundancy refers to the amount of over-specified or redundant information in the question, which can be in the form of either attributes or relationships. For example, in Fig. 1, “what color is the large bus behind the cyan car”, large (attributes) and behind the cyan car (relationship) are redundant because there is only one bus in the image. As observed in linguistics and cognitive science [12, 54, 49, 33], human speakers may include over-specified information when identifying a target object, which has also been studied in referring expression generation [46]. For VQA, as analyzed in [39], a significant difference between synthetic and real datasets is that real questions contain some redundant information, which sometimes is a distraction leading to model prediction errors. Therefore, in this work, we generate questions with different redundancy levels and study the effect of question redundancy on model behaviors.

Concept distribution. The distributions of concept, i.e. objects (e.g. car) and attributes (e.g. large), are distinct across different VQA datasets. For example, while colors are well-balanced in CLEVR dataset, in the GQA dataset, the color distribution is long-tailed where “white” appears $>50$ times more frequently than “gold”. Long-tailed distributions have been a challenge in many computer vision tasks [42, 64, 20, 56, 28]. In VQA, the long-tailed concept distribution not only hinders the learning of infrequent concepts due to few training samples, but also introduces strong biases and priors in the dataset that may mislead the models. For example, “tennis” is the correct answer to most questions with “what sport is …” [2]. With strong priors in data, it is hard to assess the true reasoning capacity of current models. While previous works address this problem by carefully re-balancing datasets [2], in our work, we controllably vary the concept distribution in our dataset and study model robustness to concept distribution shifts.

Concept compositionality. Concept compositionality refers to how different concepts (shapes, attributes) compose and co-occur with each other, e.g. roses are usually red while violets are usually blue [30]. Concept compositionality can be viewed as a conditional concept distribution in the context of other concepts. Shifts in concept compositionality impede the generalization of VQA models. For example, the model may fail to recognize a green banana because most bananas are yellow in the training data [7]. Previous works evaluate the out-of-distribution performance by collecting counterfactual testing examples [47]. In our work, we control the compositionality of shapes and colors with an intuitive motivation: if, for example, in the training data, bicycles are red and cars are blue, will the models be able to recognize blue bicycles and red cars in testing?

Super-CLEVR follows a similar data generation pipeline as CLEVR, but with more complex visual components and better control of domain gap factors. We describe the generation procedure below.

Objects with parts. To improve the visual complexity of CLEVR scenes, we replace the simple shapes (e.g., cube, sphere) in CLEVR dataset with vehicles from UDA-Part dataset [40]. There are 21 vehicle models, belonging to 5 categories: car, motorbike, aeroplane, bus, and bicycle. Each 3D model comes with part annotations, e.g., left front wheel or left right door for car. Examples for the vehicle models are shown in Fig. 2. We remove or merge small parts from the original annotations to avoid severe difficulty in visual understanding. The full object and parts list is in the supplementary material.

Attributes. Besides the attributes in the original CLEVR dataset, i.e. color, material, size, we optionally add texture as an additional attribute to increase visual complexity. Note that in order to enable part-based questions, the attributes (color or material) of object parts can be different from that of the object. For example, a blue car can have a red wheel or a green door. In this case, the attribute of the holistic object refers to the attribute of its main body (e.g. the blue car has blue frame).

Scene rendering. Following CLEVR, each scene contains 3 to 10 objects. The objects are placed onto the ground plane with random position and orientation. When placing the objects, we ensure that the objects do not overlap with each other and we avoid severe occlusion by thresholding the number of visible pixels for each object. Random jitters are added to lamp and camera positions. When rendering, we also save the ground-truth bounding boxes and segmentation masks for each of the objects and their parts, which are required when training some of the models.

Question generation. Super-CLEVR follows similar question generation pipeline in CLEVR, which instantiates question templates using the underlying reasoning program that can be operated on the scene graph. For example, the program select_shape(truck) $\to$ query_color($\cdot$) can be instantiated as question “what is the color of the truck”. Therefore, redundancy level of questions can be controlled by removing or adding redundant reasoning steps in the underlying reasoning program.

To study domain generalization, we generate several variants of the dataset for each of the domain shift factors. The variants of the datasets serve as different data domains to test the model robustness. Here we describe the method for controllably generating the dataset variants.

Visual complexity. We generate three variants of the dataset with different levels of visual complexity: easy, mid (middle) and hard. The only difference between the 3 versions is visual complexity: for the easy version, objects with different sizes, colors and materials are placed into the scene; for the middle version, we choose 3 parts on each object that are visible and randomly change their attributes; for the hard version, we further add random textures to the objects and parts. An example of the 3 dataset versions can be found in Fig. 1. Note that the scene layout and the questions are shared, so that the influence of visual complexity can be isolated and studied independently.

Question redundancy. Three variants of the dataset with different redundancy levels are generated: rd-, rd (default), rd+. By default (rd), as in original CLEVR dataset, the questions contain some redundant attributes resulting from random sampling, while all redundant relationships are removed. In rd-, we also remove all redundant attributes from the questions, leading to no redundancy in the questions. In rd+, we add all possible attributes and relationships into the question, so that questions contain a high level of redundancy. For all the variants, the questions are ensured to be valid.

Concept distribution. We generate three dataset variants with different concept distributions: bal (balanced), slt (slightly unbalanced) and long (long-tail distributed). More specifically, we change the distribution of shapes, colors and materials while the distribution of size is kept fixed in order to keep visual complexity consistent, since objects with smaller sizes are visually harder to recognize. By default (bal), the shapes and attributes are randomly sampled, leading to a balanced distribution. For slt and long, the concept distribution $\mathbf{d}$ is generated by $d_{i}=a^{-i}$, where $i$ is the index of the concept. $a$ is a hyper-parameter controlling the length of the tail. A larger $a$ leads to more imbalanced distribution and $a=1$ leads to flat distribution (cf. Fig. 1). For slt, $a=1.3$; for long, $a=2.0$. In addition, to better analyze the performance on the frequent and rare concepts, we generate three variants for testing purpose only: head (frequent concepts in the long-tail distribution), tail (infrequent/rare concepts), and oppo (opposite to the long-tail distribution). We test each model on those three variants to analyze the performance on concepts with different degrees of frequency.

Concept compositionality. We generate 3 versions of the dataset, co-0, co-1 and co-2, with different compositions of the 21 shapes (from 5 categories) and the 8 colors. The compositionality of the dataset is controlled with the co-distribution matrix $M\in\mathbb{R}^{21\times 8}$, where each entry $M_{ij}$ is the probability an object of the $i$-th shape has the $j$-th color. Entries in each row of $M$ sum up to $1$. In the version co-0, $M$ is a flat matrix so that the shapes and colors are randomly composed. In co-1, each shape in one category has a different color distribution, e.g. truck and sedan, while shapes from different categories may share the same color distribution e.g. sedan and airliner. Oppositely, in co-2, we make the shapes in same category have the same color distribution, whiles shapes from different categories have different distributions. The motivation is that since shapes from the same category are visually similar, the difference in co-1 and co-2 will help analyze the difference in model predictions on visually similar objects and dissimilar objects when composed with different color distributions.

Dataset Statistics. Every dataset variant contains 30k images, including 20k for training, 5k for validation and 5k for testing. Each image is paired with 10 object-based and 10 part-based questions. By default, the dataset refers to the version with mid visual complexity level (i.e. objects are untextured and has up to 3 parts with distinct attributes), rd redundancy level, balanced (bal) concept distribution and random (co-0) compositionality. More dataset statistics are in supplementary materials.

Random, Majority. These simple baselines pick a random or the most frequent answer for each question type in the training set as the predicted answer.

LSTM. This question-only baseline encodes the question word embeddings with LSTM [21] and predicts answer with an MLP on top of the final hidden states of the LSTM.

CNN+LSTM. The image is represented with features extracted by CNN and question is encoded by LSTM. An MLP predicts answer scores based on the concatenation of image and question features.

FiLM. We choose Feature-wise Linear Modulation [50] as a representative of classic two-stream feature merging methods. The question features extracted with GRU [10] and image features extracted with CNN are fused with the proposed FiLM module.

mDETR. mDETR [29] is a transformer-based detector trained to detects objects in an image conditioned on a text query. The model is pretrained with 1.3M image and text pairs and can be finetuned for various downstream tasks like referring expression understanding or VQA.

NSCL. The Neuro-Symbolic Concept Learner [45] is a representative neural symbolic method. NSCL executes neural modules on the scene representation based on the reasoning program, during which the modules learns embeddings of each concept with the answer supervision.

NSVQA. Neural-Symbolic VQA [59] is a neural symbolic method composed of three components: A scene parser (Mask-RCNN [19]) that segments an input image and recovers a structural scene representation, a question parser that converts a question from natural language into a program and a program executor that runs the program on the structural scene representation to obtain the answer. Notably, compared to NSCL, the individual components of NSVQA can be learned separately, hence, for example the scene parser can be learned from data that does not necessarily have Visual-Question annotations.

Since the program executor in NSVQA is a collection of deterministic, generic functional modules, it can be augmented with a probabilistic reasoning process that takes into account the confidence of the predictions of the scene parser. This allows the model to execute the program that has the largest joint likelihood, instead of only taking the maximal likelihood execution at each step of the program. The experiment results demonstrate a significant performance improvement of this probabilistic approach over the deterministic NSVQA model proposed in [59].

In particular, we interpret the confidence of the Mask-RCNN output as a likelihood function for all detected object classes $p_{object}$ and their attributes $p_{att}$. Moreover, we define a likelihood $p_{spatial}$ for the spatial relations between objects (behind, in front, left, right) that is proportional to the distance between the centers of two bounding boxes. Given a reasoning program containing multiple reasoning steps, we execute each step based on the scene parsing likelihood and produce an step-wise output with confidence. Finally, we use a factorized model, multiplying the output for all the steps to get the final answer prediction. We refer readers to the Appendix for more details.

Training mDETR requires ground-truth grounding of question tokens to image regions, which is available in Super-CLEVR. NSCL requires bounding box of objects, which can predicted using a trained Faster RCNN, and the reasoning program, which can be parsed using a trained parser. Similarly, ground-truth programs are used for training NSVQA and P-NSVQA. Note that we empirically find that the question-to-program parsing is a relatively easy task ($>99\%$ accuracy using a simple LSTM), so we focus more on models’ reasoning ability in our analysis.

Unless specified, the models are trained with default setting as in the official implementation. FiLM is trained for 100k iterations with batch size 256. mDETR is trained for 30 epochs with batch size 64 using 2 GPUs for both the grounding stage and the answer classification stage. NSCL is trained for 80 epochs with batch size 32. For NSVQA and P-NSVQA, we first train the object parser (Mask RCNN [19]) for 30k iterations with batch size 16, then train the attribute extraction model (using the Res50 backbone) for 100 epochs with batch size 64. For P-NSVQA, when counting the objects or determining whether objects exist in the scene, we use a threshold (0.7) to obtain the final selected objects. Early stopping is used based on validation accuracy. All the models are trained with 200k questions. We repeat experiments on default split for 3 times with different random seeds and get std of $\pm 0.10$ (P-NSVQA) and $\pm 0.40$ (NSVQA), showing the statistical significance of our results, then we only run other experiments once.

In-domain evaluation refers to the setting where the training and testing data come from the same domain (the default dataset variant in this case). We compare the in-domain results on Super-CLEVR and CLEVR. The results are shown in Fig. 3.

For all the models, the performance is lower on Super-CLEVR than CLEVR, suggesting that Super-CLEVR is a more challenging benchmark. The scenes with vehicles are much harder visually for the models to understand compared with the simpler shapes from CLEVR. Note that the performance gap on two datasets for simple baselines (Random, Majority, LSTM, CNN+LSTM) is smaller than for the other better models. This is because Super-CLEVR contains more object types, and therefore the performance of simply guessing is lower than on CLEVR.

When comparing the performance of different models, we find that the neural modular methods, i.e. NSCL, NSVQA, P-NSVQA, perform much better than non-modular ones. This is not surprising given their nearly perfect performance on the original CLEVR dataset, which shows its strong ability to model synthetic images. The large-scale pretrained grounding model mDETR, which is a leading model on both real and synthetic images, also achieves good performance (82.7%) on Super-CLEVR. The two-stream method FiLM does not achieve very strong performance (53.2%), but is still much better than the other simple baselines.

Our proposed P-NSVQA outperforms all the other models. In particular, on Super-CLEVR, it outperforms its deterministic counterpart, NSVQA, by 2.96%. This shows the advantage of taking into account the probabilities when the scenes are challenging thus the model’s uncertainty of predictions can be utilized.

In this section, we train and test the five models (FiLM, mDETR, NSCL, NSVQA and P-NSVQA) on different dataset variants, and diagnose their domain robustness on each of the four domain shift factors. Please refer to Sec. 3.3 for a description of different variants. The validation accuracy is used for analysis here and the results are shown in Tab. 1.

All the methods suffer from domain shifts. The results show that the best performance mostly occurs in situations where the model is tested on the same dataset variant as it is trained on, i.e. the bold or underlined numbers fall mostly on the diagonals in Tab. 1.

We compare the domain robustness of the five models by measuring the relative performance decrease when the testing data differs from the training data, i.e. smaller performance drop on different testing domains means better robustness. Based on this intuition, for easier understanding of Tab. 1, we propose a measurement metric for domain robustness named Relative Degrade (RD) to better analyze the results. We define Relative Degrade as the the percentage of accuracy decrease when the model is tested under a domain shift, i.e. the accuracy drop divided by the in-domain accuracy. Specifically, if a model gets accuracy $a$ under in-domain testing (i.e. testing with the same dataset variant as training) and accuracy $b$ under out-of-domain testing (i.e. testing with a different dataset variant from training), then $RD=(a-b)/a$. Since we train each model on three data variants, the $RD$’s for the three models are averaged to measure its domain robustness.¹¹1For concept distributions, we compute relative degrade with a slight change: we compute the accuracy drop from head to tail and the drop from long to oppo, take their average, and divide by the accuracy on bal.

Tab. 2 shows the Relative Degrade of the five models on the four factors. We see that P-NSVQA outperforms other models by a significant margin on three of the four factors, indicating that it has better overall domain robustness. In the following, we take a closer look at the results on each of the factors separately, to diagnose the influence of different model designs.

Question redundancy. Neural modular methods are much more robust to question redundancy shifts than non-modular ones. The relative degrades for modular methods are less than $2\%$, while one-modular ones degrade for around $20\%$. Due to the step-by-step design of the reasoning in modular methods, each reasoning step is independent of the others so that the models are less likely to learn the spurious correlation between question and answers. Therefore the modular methods are less vulnerable to change in question/program length.

Visual complexity. Different from our findings on question redundancy, for domain shifts in visual complexity, non-modular methods are more robust compared to modular ones. As shown in Tab. 2, while FiLM and mDETR gets less than $10\%$ degrade, NSCL and (P-)NSVQA degrade for more than $12\%$. The reason might be that the simple reasoning modules in modular methods can not process the visual signals as well as the dense non-modular models.

Comparing P-NSVQA with NSVQA, we find that injecting probability into deterministic symbolic reasoning greatly improves the robustness on visual complexity ($4.04\%$ decrease in RD). This suggests that some errors in visual understanding can be corrected and recovered by taking into account the uncertainty of visual parsing and combining the results of each reasoning step with probability.

Concept distribution. While all the four existing models suffer a lot (larger than $20\%$ RD) on domain shifts in concept distribution, we see that the symbolic method NSVQA is better than the other three (by more than $7.5\%$). With the disentangled reasoning and visual understanding components in NSVQA, the distribution priors in the images and the programs/answers cannot intertwine with each other, which prevent the model heavily relying on the priors. With uncertainty, we can further boost the robustness of NSVQA with a large margin (from $21\%$ to $14\%$ RD).

Moreover, the head-tail results suggests that the overall accuracy, which is commonly used to measure VQA performance, should be taken with cautious. When the testing split is imbalanced, the seemingly high accuracy is misleading because the head concepts dominates the testing while the tail ones are not well-reflected. For example, for NSCL, although it gets high accuracy (91%) on the long-tailed data, its performance is only 60.6% on the tail concepts. In real-world datasets, the data are usually not well-balanced, which suggests the value of synthetic testing.

Concept compositionality. Comparing the existing methods, we find that the non-modular methods seems to be more robust than modular methods NSCL or NSVQA. However, with uncertainty, P-NSVQA improves the result of NSVQA, which even outperforms the non-modular methods. This suggest the large potential of better robustness of modular methods by improving current models.

In summary, while non-modular methods are more robust to visual complexity shifts, the modular symbolic methods (improved with uncertainty) are more robust on the other three factors. By disentangling reasoning with visual understanding, separately executing every each reasoning step then merging the results of the steps using probabilities based on uncertainty, our P-NSVQA outperforms all the existing models in question redundancy, concept distribution and compositionality. Therefore, we suggest that symbolic reasoning with uncertainty leads to strong VQA models that are robust to domain shifts.

Synthetic-to-real transfer. We provide an additional proof-of-concept study to show that the findings drawn from Super-CLEVR dataset can transfer to real datasets. In the following experiments, we show our finding that neuro-symbolic methods (NSCL, NSVQA, P-NSVQA) are more robust than mDETR on question redundancy also holds true on the real GQA dataset [25]. More precisely, we progressively removed the redundant operations from the reasoning program in GQA testdev split, and then regenerated questions using a program-to-question generator. Using the change of models’ testing accuracy as the redundant operations are removed, we can evaluate the models’ robustness towards question redundancy. The results are show in Tab. 3.²²2For implementation of NSCL on a real-word dataset, we use the model in [39] (the version without calibration). The model accuracies on the original not-perturbed GQA testdev split are as following: mDETR (61.67%), NSCL (56.13%), NSVQA (39.58%), P-NSVQA (39.66%). We observe that the performance drop of mDETR is much larger than neuro-symbolic methods as the redundant information is progressively removed, which indicates that symoblic methods have better question redundancy than mDETR on GQA dataset. This is consistent with our findings on Super-CLEVR.

Reasoning with part-object hierarchies. In addition to evaluating domain generalization, Super-CLEVR can be extended for broader purposes, e.g. part-based reasoning. We can ask questions like “what is the color of the front wheel of the bike?”, “what is the color of the vehicle that has a yellow wheel”, etc. Those questions require the model to correctly understanding the part-object hierarchy, which is an ability that current VQA models lack.

Limitations. The main limitations of our work lie in the synthetic nature of our dataset. Future efforts can be made in collecting better controlled and balanced real datasets for model diagnosis. We emphasize that the purpose of the dataset is for model diagnosis and that models should also be tested on real data.