Second Place Solution of WSDM2023 Toloka
Visual Question Answering Challenge

Recent developments in VLP [26, 25, 15, 5, 18, 19] have been propelled by pioneering models such as OFA [14] and BLIP [6]. These models have innovatively utilized large-scale datasets to pretrain transformer-based architectures on paired image-text data. OFA aims to unify various vision-language tasks by leveraging comprehensive pretraining, thereby enhancing performance in applications such as VQA and image-text retrieval. Meanwhile, BLIP focuses on optimizing data efficiency and model performance through advanced self-supervised learning techniques, contributing significantly to the robustness of multimodal understanding. In this evolving landscape, ChatGPT-4 [11] has emerged as a notable advancement, showcasing remarkable capabilities in generating nuanced natural language responses and pushing the boundaries of language modeling tasks.

VQA [4, 2, 13, 22] involves answering questions about images using natural language understanding, which has been a pivotal area of research in multimodal AI. These tasks require models to comprehend both the visual content of images and the semantic context of textual queries. Early approaches focused on integrating vision and language through methods like joint embeddings and attention mechanisms. More recent advancements, leveraging models such as OFA [14] and BLIP [6], have significantly improved the accuracy and efficiency of VQA systems by employing large-scale pretraining on diverse datasets. These models aim to enhance the model’s ability to reason about complex scenes and provide accurate responses based on multimodal inputs. VQA continues to evolve with the introduction of transformer-based architectures like ChatGPT-4, which further pushes the boundaries of multimodal understanding and response generation in AI systems.

Visual grounding [7, 12, 24, 1, 9, 23] tasks involve the localization and identification of specific objects or regions within an image based on textual descriptions. This task is crucial in multimodal AI applications, where models need to link natural language queries to corresponding visual elements effectively. By leveraging deep learning techniques, such as convolutional neural networks (CNNs) for image feature extraction and transformer architectures for textual understanding, models can align visual and textual modalities. This alignment enables precise object localization, scene understanding, and context-based reasoning, enhancing applications like VQA [4, 2], image retrieval, and interactive systems that require accurate interpretation and response generation based on visual content.

The first part is the coarse tuning stage. We construct the coarse-tuning dataset through coco images. There are several features of constructing synthetic datasets. First, the image distribution, question distribution, and bounding box distribution of the coarse tuning dataset should be as close as possible to the competition dataset, so that the model can avoid learning the wrong data distribution; Secondly, any sampling operation must be Random Sampling, which can ensure that the model will not learn any data bias. Finally, do not sample any data in the test public set, which can avoid overfitting the model to the test public set. The coarse-tuning paradigm should be the same as the competition task, similar to the pre-training, save the coarse-tuned model weight for the next fine-tuning part. similar to the pre-training, the more coarse tuning epochs, the stronger generalization, and the better performance.

This figure illustrates our process of constructing the synthetic dataset. For each sample in the training set, we directly input images and questions into the multi-modal pre-training model to obtain textual answers, which we define as pseudo-answers. Now, each sample consists of an image, a question, and a pseudo-answer; therefore, we can build a mapping table of pseudo-answers and questions, where one pseudo-answer corresponds to multiple questions, for example, the pseudo-answers clock and vase. Then, randomly selected an image from the coco and detected several objects by the object detector. Each detected object includes confidence, class counts, class name, and bounding box, where class number is defined as the number of the object in the image, such as the number of clocks is one and the number of roll paper is two. Next, we sorted the detected objects according to confidence, selected an object class with the largest confidence, the class count is one, and belonging to the pseudo answer, then, randomly selected a question corresponding to the pseudo answer from the mapping table. Now, a new sample is obtained, the question of the sample is randomly selected question, the pseudo-answer is the selected object class, and the target bounding box is obtained by the object detector. After the synthetic dataset is constructed, a model with weak semantics but strong generalization can be coarse tuning in the form of visual grounding.

The second part is the Fine-tuning stage. In the form of a visual grounding task, load the trained weight obtained by the last Coarse tuning part and continue to train the model on the competition dataset. Each sample consists of an image, a question, and a pseudo-answer. Question and pseudo answer are combined into the template, which is used as text input of the model. For the four templates we constructed, we finally adopted the second template, and the first word of the question, such as what, where, and which, was replaced with which region. After the template construction, the template and image are input into the OFA model, through text encoder, image encoder, and cross-modal fusion, the coordinates of the bounding box are finally output by text decoder.

The third part is the post-processing stage, the bounding box position predicted by OFA is roughly correct, but the accuracy of coordinates is not higher than object detector. Therefore, by calculating the iou, we replace the predicted bounding box with the object detector’s bounding box. For each image, all bounding boxes in the image are detected by object detectors yolor and vitDet, and sorted according to the confidence, which are called candidate Bounding boxes. Then, for the predicted bounding box, the first candidate bounding box with iou higher than zero point six is selected to replace the predicted bounding box. Finally, the model ensemble includes three ways, the first is five folds with standard Corase tuning, the second is ten folds with standard Corase tuning, and the third is five folds with back translation Corase tuning. The back translation was applied to the coarse tuning stage, which expanded the diversity of question data.

The Toloka dataset comprises images paired with textual questions, where each entry includes a question-image pair annotated with ground truth bounding box coordinates pinpointing the visual answer. In total, the dataset consists of 45,199 instances distributed across three subsets: 38,990 instances in the training set, 1,705 instances in the public test set, and 4,504 instances in the private test set.

The dataset is structured with several key columns: ”image” contains URLs linking to images hosted on a public content delivery network; ”question” provides English-language queries associated with each image. Additional metadata includes ”width” and ”height” integers representing the dimensions of each image. For bounding box annotation, the dataset includes ”left,” ”top,” ”right,” and ”bottom” integers detailing the coordinates that define the spatial extent of the object or region in the image that corresponds to the answer to the question.