DALL-E for Detection: Language-driven Compositional Image Synthesis for Object Detection

Training modern deep learning models require large labeled datasets [38, 22, 44]. Obtaining such datasets is both expensive and time-consuming due to large human effort requirements. This raises a question: can we efficiently generate a large scale labelled data that also achieves high accuracy on a new downstream task? We first hypothesize that any such approach should satisfy these qualities (Table. 1): no (or minimal) human involvement, automatic generalization of the images for any new classes and environments, scalable, generation of high quality and diverse set of images, explainable, compositional, and privacy-preserving.

To this end, synthetic techniques could be used as possible sources for generating labelled data for training computer vision models. One popular approach is to use computer graphics to generate data [41, 45, 48]. However, these approaches may require gathering 3D models of both objects and scenes, which can require a large amount of skilled labor, such as 3D modeling expertise, which prohibits the scalability of such graphics generated synthetic labelled data. Another approach is to use object-cut and paste [12], which is a 2D synthetic generation approach, but these approach can have limitations as they still require a source for foreground objects and accurate foreground masks for those objects. A third approach could be to use machine learning based neural renderer techniques like NeRF based approaches [33, 15]. These approaches generally require retraining models for any new object class and so can not easily scale to large number of object classes.

Recently, there has been a revolution in large scale text-to-image synthesis models like DALL-E [37], RU-DALLE [1], CogView [8], and Stable Diffusion [42]. They have been shown to achieve photorealism even for complex scenes, able to understand semantics and compositional nature of the real world. In addition, these models can understand language descriptions of a scene and so can act a natural bridge between human and synthesis approaches. Given these qualities, can these text-to-image synthesis approaches be used to generate large scale training data with accurate labels for computer vision problems? For simplicity, from now on, we will use DALL-E to represent text-to-image synthesize models (including Stable Diffusion and RU-DALLE).

In this work, we propose a new text-to-image synthesis paradigm involving two components to generate large scale training data with accurate labels for tasks like object detection and instance segmentation. First part involves accurately generating foreground object masks for the object classes of interest. This is due to the reason that DALL-E can not generate annotations, e.g., they are not able to generate bounding box for the objects. In order to generate diverse foreground object masks, we first generate images containing mostly one object corresponding to the interest class. We use a simple template with class name as input to the DALL-E pipeline. For example, in order to generate images with cat, we use input as an image of cat on pure background. Next, a simple background-foreground segmentation approach is used to get foreground object masks of the interest classes.

The second step involves generating diverse background images that provide good context information for training recognition models. Context plays an important role in learning a good object recognition model. Divalla et al. [9] provide empirical evidence to support this claim. Dvornik [11] showed that finding congruent context helped improve accuracy on object detection tasks. For example, placing airplanes and boats in their natural context helped to improve accuracy, e.g., airplanes are generally found in the sky and boats are on the water.

We again leverage DALL-E to generate diverse set of high quality context images. At the core of our approach lies utilizing an interplay between language description of context and language-driven image generation. Given a small number of images that represent the context environment, we use image captioning to generate a high-level language description of the context automatically. The language description of the context is used within a text-to-image generation pipeline (e.g., DALL-E) to generate a diverse set of images. These diverse sets of generated images are used as context images.

Finally, to generate labelled data, we follow a simple strategy where we paste the foreground object masks (from step one) onto the random context images (from step two), as in object-cut and paste approaches [12]. The proposed pipeline satisfies all the desired properties of labelled image generation (Fig. 2). The data can be generated efficiently without human involvement effortlessly. Language descriptions for both the foreground and the background image generation help to provide an explainable and compositional data generation. Adding or removing objects or settings can be easily done in the language domain. For example, a description as an environment with a table can be easily modified to kitchen environment by utilizing the compositional properties of language as a kitchen environment with a table. Table. 1 shows the benefit of our approach in generating context images over other approaches.

We have conducted extensive experiments on four publicly available benchmark object detection datasets and compared them against different ways to generate context images. We demonstrate that our approach can achieve much better accuracy compared to the prior approaches. We also demonstrate the benefit of our approaches in out-of-distribution context and zero-shot data generation scenarios that utilize the compositional nature of our method. Our main contributions are: (1) We propose a language-driven compositional image generation approach to automatically generate both foreground objects and background context images and form large-scale datasets. (2) We demonstrate benefit of our proposed pipeline over several prior approaches. (3) We highlight compositional nature by generating context images from out-of-context images and zero-shot data generation scenarios. To the best of our knowledge, this is the first work to use vision and language models for generating object detection and segmentation datasets.

The goal of the paper is to efficiently generate a large set of labeled data for training object detection models using text-to-image synthesis frameworks. In particular, the proposed approach decouples training data generation into diverse set of foreground object mask generation and diverse set of background (context) image generation. The foreground object masks are then composited onto the background images following the object cut-and-paste strategy [12]. Furthermore, the proposed approach also allows for compositional and explainable data generation as well. Our pipeline leverages off-the-shelf language generative frameworks [24, 42] under textual guidance [25] to generate both the foreground masks and context images.

We demonstrate the effectiveness of the proposed approach in automatically generating large datasets on three scenarios. First we evaluate our method on large scale object detection tasks on Pascal VOC and COCO datasets. Here, large scale training data has been created by synthetically generating the foreground masks for both VOC and COCO object classes, and background (context) images as discussed in the method section. Next, we highlight the benefits our approach on instance detection tasks. We consider GMU-Kitchen [17], Active Vision [3], and YCB-video datasets [53]. We follow the training strategy of [12] and since these datasets already provide object masks, we use our method to only generate good context images for downstream object instance detection tasks on the above three datasets. Finally, we also provide results highlighting compositional nature of our data generation process. For Pascal VOC and COCO dataset experiments, we use Stable Diffusion [42] as the text-to-image generation model, and for other experiments, we use RU-DALLE [1].

We have proposed a new paradigm to generate large scale labelled data for object detection and segmentation tasks using large vision and language-based text-to-image synthesis frameworks. We demonstrate effortless labelled data generation on popular benchmarks for object detection tasks. Computer vision models trained using these data improves the performance over the models trained with large real data. Thus reducing the need for expensive human labeling process. We also highlight the compositional nature of our data generation approach on out-of-distribution and zero-shot data generation scenarios.

There are interesting extensions. Our approach can be used to solve other tasks like instance segmentation task, handling long-tail and imbalanced data problems. Next, the presented approach can be used to make models robust by iteratively generating the data where the detection model fails, which are then used to improve the robustness of the model. We believe the presented ideas of using vision-text models for automatically generating large labelled data opens door to democratizing computer vision models.

We provide details of our methodology in Section 6, including ablation studies of various components in our pipeline and some observations on those components’ importance. Further we show that our pipeline can naturally be extended to other tasks such as low resource instance segmentation in Section 7. Moreover, we demonstrate that blending real data can further boost the performance in Section 8. Lastly, in Section 9 we show that by utilizing compositionality of natural language we can narrow the domain gap and tackle out-of-domain problem. For completeness, we give model prediction in Section 10.

We provide more details of our synthesized dataset here. Consider VOC [13] dataset with 20 objects²²2Synthetic dataset for other datasets like COCO [31] is generated similarly..

For context background generation, approach 1 uses CDI to provide context description. Specifically, $K=2$ captions are generated for each of the CDI, and each caption generates $M=80$ images. As a post-processing step, we use CLIP [36] to filter $M$ images such that only $30$ context images remain. Therefore for each CDI we generate $K\cdot M=2\cdot 30=60$ images. In 10 shot we have a total of $20\cdot 10\cdot 60=12,000$ synthesized context images; while for 1 shot we have a total of $20\cdot 1\cdot 60=1,200$ context images. For approach 2, we design 16 templates, and 600 images are generated for each template. We observe that generated images are of high quality and generally do not contain interested objects. Therefore we prune 5% images by CLIP only, which results in 9,120 synthetic backgrounds.

For synthesizing images for the foreground objects (Section 6.2), we generate 500 images for each of the $6$ templates and also use CLIP [56] to select 250 best images. Thus we have a total of $20\cdot 6\cdot 250=30,000$ synthesized foreground images for the 20 objects in VOC. Since foreground mask extraction process will drop some bad examples by design, the number of final processed extracted foregrounds is $26,685$. We repeat cut-and-paste such that the final pasted dataset have size $60,000$. Specifically, in each step, we first randomly select a background image, then 4 randomly selected foreground masks are pasted at random locations in the background image. Note that our generated dataset contains no duplicate.

Fig. 6 shows more example training images generated by our pipeline on PASCAL VOC dataset: both foreground object and background context images are generated by our method with Stable Diffusion.

Although we mainly discuss object detection task in the main paper, we highlight that our method is not restricted to detection tasks only. In this section we demonstrate the effectiveness of our pipeline in instance segmentation. This is a simple extension since during the paste process we obtain not only bounding box but also segmentation task.

We report the performance in Table 9. Similar to experiments we layout in the main paper, we apply our method in low resource setting on PASCAL VOC [13] dataset. We conduct experiments on 10 shot and 1 shot. We observe a strong performance in both low resource settings, in which our method significantly outperforms the direct supervision (+30.69% improvement on mAP@50 in 10-shot setting). In 1 shot scenario in which direct supervision can not make any correct prediction, our method can achieve 46.74 mAP@50.

We first demonstrate the effect of incorporating different percentages of real-world training images together with our synthesized images for training object detection models. We conduct experiments on the GMU kitchen [17] test set and the real-world training images are from the GMU kitchen training dataset (100% set contains 3837 images). This experimental setup is similar to the one followed in the Object cut-and-paste paper [12]. All the results are provided in the Table 10. We highlight the mAP accuracy of training with all synthetic data plus 10%, 40%, 70% and 100% real data.

Observe how using only a subset of real-world data ($70\%$) with our synthesized images achieves better performance than full (100%) real-world data only. This suggests the advantages of our data generation approach saving the amount of human efforts required in labeling the real-world data significantly. Further, we also observe that accuracy gradually improves from $78.3\%$ to $91.4\%$ as we increase the amount of real-world data.

Here we provide some visualization examples of context images generated from just one given Context Description Images (CDI). To demonstrate that our model is very generic and can generate a large set of diverse context images from given input as little as one image, we include some examples of generated images from our pipeline as shown in Fig.  5.

In this section, we demonstrate more results for the compositional experiments present in section 4.3 of the main paper.

In table 6 of the main paper, we evaluate the models with synthetic data generated before intervention and after an intervention. Here in table 12, we provide additional results of training models just on the CDIs. We observe that the model can not yield good results due to the insufficient amount of training instances and the large domain gap. However, if we apply our method without intervention, we can get a significant performance boost by providing diverse training instances. Furthermore, by applying intervention, we can narrow the domain gap and yield even more performance gain, reinforcing the effectiveness of our approach. For instance, Fig.8 visualize the context images generated by DALL-E for noisy and out-of-distribution CDIs. In the left column, inputs CDIs are cartoon kitchen which provide noise information about environment. The compositional property allow us change the style from cartoon to real and generate high-quality context images (right column)

Here we show qualitative examples of out-of-distribution images with their corresponding captions and how the compositional property of our method can help to correct context descriptions by remove/add and style change.

Specifically, we include figures that exemplify our generated images in the compositional experiments. Fig. 8, fig. 9, fig. 10, and fig. 11 are generated results before and after intervention for the experiment Cartoon Kitchen, Skeleton Kitchen, Objects in Kitchen, and Kitchens with Human in the Table. 12, respectively.

Fig. 12 demonstrates the predictions of faster RCNN [38] models on PASCAL VOC [13]. The model was trained with the setting of main paper Table 2 G-2 EXP-5. Fig. 13 demonstrates the predictions of faster RCNN on the GMU-Kitchen dataset. The model was trained with # CDI = 1500 from UW dataset [17]. Although the model is trained on synthetic data, the model is able to yield good predictions without being accessible to GMU Kitchen’s train data.