Renaissance: Investigating the Pretraining of Vision-Language Encoders

The domain of vision-language modeling has seen major advancements in recent years with the adaptation of the transformer (Vaswani et al. 2017a) as VL models. The first examples of VL transformers to appear in the literature were adaptations of the popular BERT NLP model (Devlin et al. 2018). Some examples include VilBERT (Lu et al. 2019), LXMERT (Tan and Bansal 2019a) and VisualBERT (Li et al. 2019a). In the short space of time since these models were introduced a bewildering array of model variations have appeared in the literature. There are huge models designed for zero-shot inference such as Flamingo (Alayrac et al. 2022) and versatile models such as OFA (Wang et al. 2022) that can generate both text and images.

While the literature is now replete with vision-language models, the analysis of their performance and the establishment of best practices has been mostly left open. The aformentioned study Bugliarello et al. (2021) examines pretraining of vision-language model. Bugliarello et al. (2023) is an effort by the same lead author, Emanuele Bugliarello, that provides an analysis of several models on what they term "fine-grained" tasks. Frank et al. (2021) examined the extent to which the vision and language modalities are actually integrated in VL transformers. As valuable as these studies have been however, they have barely scratched the surface of understanding vision-language transformers.

Vision language modeling has only recently come to prominence and the available software for it is still in a fairly primitive state. When using NLP models, researchers have a range of available software options that abstract many of the most difficult elements away from users. The most prominent example of this is the Huggingface model hub that specializes in NLP transformers. Though there are a few vision-language models available on Huggingface, there aren’t many and they don’t lend themselves to the modifications that research often demands. In addition to the Huggingface Hub, there have been some efforts toward creating software platforms primarily dedicated to multimodal modeling. LAVIS, introduced in Li et al. (2023) by Salesforce, is one such platform. Though well programmed and relatively straight forward to use, this program offers support very few VL models. Further more, implementing VL tasks is also a fairly involved task. The paucity of available software options led us to create the Renaissance platform for VL modeling that we introduce in the next section.

In this section we describe the capabilities and various options available from the Renaissance platform. The most salient feature of the platform is its ability to easily change the basic architectural features of multi-modal transformers, then train and test them. By simply editing a configuration file, a user can choose a pretrained text encoder or a pretrained vision encoder from the Huggingface hub to insert into the model. In addition to the various architectural options, there are also a number of pretraining and fine-tuning tasks and options available. We will describe these in subsections below.

Renaissance is entirely written in the Python programming language. Though popular and user friendly, using and maintaining large-scale python programs can be a difficult process. In order to make this model platform useful as a research tool, we’ve made a number of conscious decisions designed to improve the usability and versatility of the program. These design goals are discussed in the section below.

In our first set of experiments, we ask what is the effect of freezing the weights of various parts of the model during pretraining? Specifically, if we initialize the vision and text modules of a two-tower encoder with pretrained models from their respective domains, can we freeze one or both of these modules during pretraining? Freezing both modules means that we would only be pretraining the cross-modal and output layers of the model. Pretraining is usually the most compute intensive aspect of model development and we can reduce the GPU memory use and the overall compute required by freezing parts of the model. The compute savings would allow researchers to pretrain models that might otherwise be too large for their hardware requirements. Alternatively, they might also train smaller models at higher batch sizes and possibly obtain better results.

Given that both vision and text encoder modules are pretrained in their respective domains, it makes intuitive sense that we might be able to skip at least some portion of their pretraining. These experiments should demonstrate empirically whether or not this is the case. Furthermore, the creators of the dual encoder¹¹1Dual encoder models are a simpler model type that is currently not available on Renaissance. model LiT Zhai et al. (2022) found that they obtained slightly better results from freezing the model’s vision encoder. This experiment will also afford us the opportunity to see if a similar effect will hold for two-tower encoder models.

To begin, we use Renaissance to construct a set of two-tower models with ELECTRA-Small Clark et al. (2020) as the text encoder and DeiT-Tiny Touvron et al. (2021) as the image encoder. Both of these models are both quite small and efficient, and we chose them to expedite the training process. We set the cross-modal encoder module of each model to contain two sets of six transformer layers each with a hidden size of 256 and 4 attention heads. In total we pretrain four model variations, a baseline with both modules unfrozen, one with the text encoder frozen, one with the image encoder frozen and one with both encoder modules frozen. Each model will be pretrained for 100k steps at a batch size of 704 using the masked language modeling and image-text matching tasks described in Section 3.1.3. All models are trained using two NVIDIA L40s GPUs. We use two of the four pretraining datasets, MSCOCO and Visual Genome, which were described in the same section. Finally, we finetune and evaluate our models on three of the five VL tasks described in Section 3.1.4: SNLI-VE, NLVR2 and reference resolution with RefCOCO. In the interest of saving time and compute, we forego evaluating them on multimodal retrieval tasks and visual question answering. A crucial point to consider is that the no model weights are frozen during finetuning.

The results for this experiment are summarized in Table 1. We see that we can obtain similar results by freezing one or both of the previously trained encoder modules during pretraining with only mild ill effect. On the SNLI-VE task, the difference between training the whole model and freezing one or both modules is very slight indeed. When freezing the vision module, the downstream results for SNLI-VE are essentially identical and we see only a slight drop in performance compared to the baseline model. We see a slightly different pattern on the NLVR2 task, however. Here we see the results for the baseline model and two models with a single encoder frozen having almost identical results. The model with the visual encoder produces the best score on the reference resolution task. The baseline model and the model with both modules frozen preform very nearly identically, while the the model with the text encoder only frozen scores the worst. Of the four models, the overall best performance is achieved by freezing only the vision encoder.

This is a fairly remarkable result as we can essentially cut the GPU memory required during pretraining in half by freezing so many of the model’s weights. This is especially significant because pretraining the model is by far the largest compute cost we see during training. We should also note that this effect is somewhat similar to a phenomenon noted in the training of the dual encoder LiT Zhai et al. (2022) (dual encoders have two encoder stacks but lack the a cross-modal fusion module). Here Zhai et al. found that they can obtain better results from freezing the image encoder during training. Though our model architecture is different, we observe a somewhat similar effect.

To further demonstrate the utility of freezing the pretrained modules, we train a model that uses ELECTRA-Base as a text-encoder and ViT-Base as the vision encoder. We use the same training hyperparameters but increase the number of cross-attention layers to 10. This model is quite large for an encoder, containing over 210M parameters (decoder models for generative tasks often contain billions of parameters). A model of this size would be well outside our compute capacity to pretrain if the text and vision modules were not frozen. However, because it contains less than 27M trainable parameters, training it puts the same memory load on our two L40s GPUs as the baseline model does. The results for this model are displayed in the final row of Table 1. This model obtains the best results of the study on two of the three downstream tasks and remains well within our limited compute budget.

In the previous set of experiments we focused on training two-tower models. In our final experiment we will examine the behavior of one-tower models. One-tower encoder models were also described in Section 3.1.1. To date, most of these models have been derived from text encoder models such as BERT Devlin et al. (2019). A less explored approach is to base such models on transformer based vision models such as ViT Dosovitskiy et al. (2020); this is the approach of the one-tower VL transformer called ViLT Kim et al. (2021). In this experiment we ask if one strategy is superior to another when training and evaluating under otherwise similar conditions? More simply put, are one-tower encoders more effective when based on a vision encoder or a text encoder. In addition to providing guidance to future practitioners of VL modeling, answering this question should also provide interesting results from purely theoretical as well as practical perspectives.

To make this experiment as fair a comparison as possible we select a vision transformer and a text transformer model as close in size to each other and architecture as possible. Toward this end we used BERT Devlin et al. (2019) as our text-encoder model and ViT Dosovitskiy et al. (2020) as our vision encoder model. The encoder towers in each of these models were consciously designed to have nearly identical dimensions with each encoder module containing 110M parameters. We employ patch embeddings for visual tokens and word-piece embeddings for visual tokens in all models. The resultant models will be close to identical, save that the weights of one are derived from vision pretraining and the other from text/language pretraining. As a baseline, we also train a randomly initialized version based on the BERT architecture. Finally, we train each model for 50k steps with a batch size of 512 using masked-language modeling and image-text matching. Again we use MSCOCO and Visual Genome as training datasets and evaluate on the 3 vision-language tasks described in the previous experiments.

The results for this experiment are displayed in Table 2. According to our analysis there doesn’t appear to be a significant advantage in basing a one-tower encoder model on either text or vision. Surprisingly, the randomly initialized model that we trained as a baseline scored the best on all three downstream tasks. These are very much unexpected results. Though we didn’t have an intuition as to whether text or vision would perform better, we didn’t expect the downstream results to be so similar and to be inferior to a randomly initialized variation. These results are especially notable since one of the few in depth analyses of vision-language models, Frank et al. (2021), indicates that the interaction between the visual and language modalities are not symmetric. That study used probing techniques to show that VL transformers learn to use vision-for-language more than language-for-vision. Our best explanation of this phenomenon is that that one-tower models do not make use of the individual visual or textual modalities, but instead converge to values not dependent on either.

Another notable conclusion of this experiment and the preceding ones, is that two-tower models are in general much more parameter efficient than one-tower models. The one-tower models used in this experiment are relatively large, each containing more than 100M parameters. While the two-tower models in the previous experiments contain less than 40M parameters. Nonetheless, the two-tower models outperform those in this final experiment using the same datasets for training and evaluation. In previous studies, one-tower models such as ViLT Kim et al. (2021) that have similar architectures, have obtained better results than those displayed here. They do so by using more data and enormous pretraining batch sizes that require significantly more compute than we used here. Though only a preliminary finding, this insight might prove valuable to those interested in efficient VL modeling.

As this program evolves, we hope to incorporate a number of additional features that are not available in the current version. The capabilities that we plan to add are discussed below.

Because of the field of vision-language is rapidly evolving there are many possible future directions for research. We will mention a few. Though we have touched on some of the more basic aspects of training in this study, a systematic study of how each pretraining task contributes to downstream performance would be very illuminating. As would testing other tasks, such as visual grounding or visual question answering in pretraining. A thorough investigation of which architectures are best used in which circumstances would also be a worthwhile endeavor. As a final suggestion, we believe the field would also benefit from a scaling study to determine the optimum amount of data to train models at various scales such as Kaplan et al. (2020) performed for NLP transformers.