Look Ahead or Look Around?
A Theoretical Comparison Between Autoregressive and Masked Pretraining

Taking an unsupervised language dataset ${\mathcal{D}}=\{x_{i}\}_{i=1}^{N}$ as an example, each sample $x_{i}=(x_{i,1},\cdots,x_{i,s})$ consists of $s$ tokens. The objective of autoregressive SSL is to predict the subsequent word based on prefix texts, i.e., minimizing the following Negative Log-Likelihood (NLL) Loss:

where $\Theta$ are the weights of the neural network.

During the masked SSL pretraining process, for each sample $x_{i}=(x_{i,1},\cdots,x_{i,s})$, we draw a random mask $m\in\{0,1\}^{s}$ (drawing 0 with probability $\rho_{m}$, i.e., the mask ratio) and apply the mask to generate the unmasked text $x^{1}_{i}$ and masked text $x^{2}_{i}$:

The objective of masked SSL is to predict the masked tokens from unmasked texts, i.e.,

where $x^{1}_{i,k},x^{2}_{i,k}$ respectively denote the $k$-th token of the unmasked text $x^{1}_{i}$ and masked text $x^{2}_{i}$.

In practice, both the representative autoregressive SSL model GPT (Radford et al., 2018) and the representative masked SSL model BERT (Devlin et al., 2019) use the multi-layer Transformer to generate the predicted distribution and apply the Cross-Entropy (CE) loss. Let $f(X_{i})\in\mathbb{R}^{t}$ be the output of the network and $W\in\mathbb{R}^{t}\times\mathbb{R}^{N_{D}}$ be the token embedding matrix that transforms the output features to the predicted distribution of $N_{D}$ different tokens. Then the predicted probability in autoregressive and masked SSL can be uniformly reformulated as:

where $X_{i}$ are the conditional tokens, $X_{i}^{+}$ are the target tokens and $X_{i}^{-}$ are the independent tokens. To be specific, taking the sentence in Figure 1 as an example (”Euphoria means joy, happiness and excitement”), for autoregressive SSL, $X_{i}$ represents the prefix tokens (e.g., [Euphoria, means, joy, happiness, and]) and $X_{i}^{+}$ is the subsequent token ([excitement]). For masked SSL, $X_{i}$ represents the unmasked tokens (e.g., [Euphoria, joy, excitement]) while $X_{i}^{+}$ represents one of the masked tokens (e.g., [happiness]). Besides, we use a one-hot vector $\mathbbm{1}_{X_{i}^{+}}$ to indicate the target tokens (the $X_{i}^{+}$-th position of $\mathbbm{1}_{X_{i}^{+}}$ is set to 1 ). The corresponding objective is:

We note that the main objective of generative SSL loss is to maximize the probability of target words and minimize the independent word probability. Following the same spirits, we consider the following spectral loss (HaoChen et al., 2021) to simplify our theoretical analysis:

When theoretically comparing the generalization performance of autoregressive and masked SSL pretraining, it’s challenging to analyze the overall optimal solutions for their objectives as both of them are usually seen as instance-level tasks, which hinders us from demonstrating the downstream performance of pretrained models. Inspired by the previous work (HaoChen et al., 2021) which addressed analogous challenges in contrastive learning, we consider reformulating the spectral loss and analyzing it from a matrix decomposition perspective.

We start by introducing the co-occurrence matrix $A$, which formulates the joint distribution $P_{M}(X_{i},X_{i}^{+})$ of the conditional texts and target texts as a matrix, i.e.,

And we denote $P_{C}(X_{i})$$=\sum_{X_{i}^{+}}P_{M}(X_{i},X_{i}^{+})$, $P_{G}(X_{i}^{+})=\sum_{X_{i}}P_{M}(X_{i},X_{i}^{+})$ as the marginal distributions of conditional and target texts. Based on these definitions, we establish a crucial connection: the spectral loss is equivalent to an asymmetric matrix decomposition objective on the co-occurrence matrix.

Classification tasks and content generation tasks are the most commonly adopted evaluation protocols for pretrained models (Radford et al., 2018; Brown et al., 2020; Dong et al., 2019; Devlin et al., 2019; Yang et al., 2019; Chen et al., 2020a; He et al., 2022). Therefore, in this paper, we mainly consider the classification tasks and content generation tasks for generality.

Linear classification. For the classification tasks, we assume that the labels of ${\mathcal{D}}$ can be accessed, i.e., ${\mathcal{D}}=\{(x_{i},y(x_{i}))\}_{i=1}^{N}$, where $y(x_{i})$ is the ground-truth label of $x_{i}$. For each sample $x_{i}$, we first encode it using the pretrained model $f$ and then apply a linear classifier $g$ following that to generate the predicted distribution of different labels. For ease of theoretical analysis, we follow the settings of linear evaluation (Saunshi et al., 2019), i.e., the pretrained encoder $f$ is frozen during the downstream evaluation process. Then we evaluate the prediction error

where $\hat{y}_{i}(x_{i})={\rm argmax}(g(f(x_{i})))$.

Content Generation. For the content generation tasks, we consider an unsupervised text dataset ${\mathcal{D}}_{u}=\{x_{i}\}_{i=1}^{M}$ that consists of $M$ samples, we still assume that each $x$ consists of $s$ tokens, i.e., $x_{i}=(x_{i,1},\cdots,x_{i,s})$. The downstream objective is to evaluate the following likelihood with the pretrained model weights:

The exponential of this downstream objective is also called as perplexity, which is an important metric to evaluate the language model.

In this section, we compare the generalization performance between autoregressive and masked SSL on linear classification tasks. To achieve this, we first establish a unified downstream classification guarantee of generative SSL and then compare autoregressive and masked SSL based on that.

Besides the classification tasks we analyzed above, another important evaluation task for pretrained models is the content generation ability (Radford et al., 2018; Devlin et al., 2019). Previous works have empirically shown that the autoregressive SSL models outperform masked SSL models in downstream content generation tasks (Dong et al., 2019). To further understand the advantages of autoregressive SSL models in content generation tasks, we first observe the generation ability of masked SSL models with different types of texts.

As shown in Figure 3, we observe that the masked SSL model performs particularly unsatisfactory when generating from short texts. Intuitively, it can be attributed to a misalignment between the pretraining objective and the downstream evaluation tasks. To be specific, when predicting the masked words in the pretraining process, the lengths of inputs are fixed (e.g., 15% of the texts). Consequently, the masked SSL model may struggle to accurately infer the complete texts due to limited information in the downstream generation tasks. In the subsequent analysis, we aim to theoretically substantiate this intuition. For the ease of our theoretical analysis, we consider using the linear attention as the pretrained encoder to compare the generation abilities of autoregressive and masked SSL models.

Linear Attention. The general form of linear attention is given by:

where $W^{Q},W^{K},W^{V}$ are projections.

By comparing the evaluation error on the downstream generation tasks, we obtain the following theorem.

As stated in Theorem 4.3, besides the canonical pretraining error, the autoregressive SSL models have superior performance guarantees in downstream content generation tasks compared to masked SSL models, and the performance gap is mainly decided by two crucial factors: the alignment of input lengths ($w_{k}$) and the consistency of different positions ($\eta$). In the following, we respectively discuss these two factors.

Alignment of input lengths. We note that when the lengths of test samples are close to the length of unmasked texts in the pretraining objective, the performance of masked SSL models can approach autoregressive SSL models, which proves the length misalignment between unmasked texts in pretraining and test examples in downstream is a crucial reason for the inferior performance of masked SSL models.

Consistency of different positions. The term $\eta$ in Equation (4) evaluates the difference in model output across various positions in a sequence. When the output distribution is more consistent, the performance of the masked SSL model is better. Consequently, it is advantageous to encourage the model to generate predictions based on the entire sentence rather than focusing on specific tokens. Besides, the consistency of different positions offers some other potential benefits. For example, by uniformly considering the semantic information across different positions in a sentence, the model can avoid generating shortcuts and overfit solutions.

In summary, the theoretical bounds in Theorem 4.3 provide two principled guidelines to improve the content generation performance of masked SSL: (1) we should try to mitigate the length misalignment between pretraining and downstream examples, and (2) the model predictions should be encouraged to be consistent with different tokens in the same sentences.

Inspired by the theoretical analysis, predicting the masked texts with different lengths of unmasked texts is a straightforward solution to mitigate the misalignment and improve the generation performance of masked SSL. Consequently, we propose the following variable-length masked objective:

In this objective, the mask ratio $\rho_{m}$ is not fixed. Instead, it is randomly sampled from a range. The variable mask ratio has the potential to mitigate the length misalignment between pretraining and downstream examples, improving the content generation performance. We will verify its effectiveness in Section 5.

In summary, this section introduces the first theoretical comparison between autoregressive and masked SSL. As autoregressive and masked SSL exhibit contrasting advantages in downstream classification and content generation tasks (Radford et al., 2018; Devlin et al., 2019), the two most crucial evaluation tasks for pretrained models seem to be contradictory. However, by revealing the limitations of autoregressive SSL in classification tasks (fixed position of target tokens) and limitations of masked SSL in content generation tasks (fixed length of unmasked texts), we note that the limitations of these two paradigms are not contradictory to each other. Instead, they deliver a consistent message: we should encourage diversity (in inputs, prediction objectives, etc) to the generative SSL objectives to improve the generalization performance. As a verification of our analysis, we will show the performance of our proposed objectives across both language and vision tasks in the following sections.

In this part, we will consider the diversity-enhanced autoregressive objective as proposed in Equation (3). The diversity-enhanced autoregressive objective is to predict the next $t$ tokens of conditional texts. Inspired by (Wang et al., 2018b), which predicts several tokens simultaneously, we perform group-autoregressive modeling on the sequence to effectively realize this objective. Specifically, the sample sequence $(x_{i,1},\dots,x_{i,s})$ is divided into several consecutive groups $G_{i,1},\dots,G_{i,l}$ in order, with each group containing any number of tokens. The prediction is performed group by group, i.e., within each group, the tokens are predicted in parallel, while across group, the predictions are sequential. In this way, given $G_{i,1},\dots,G_{i,s}$, the model is to predict all tokens from $G_{i,s+1}$. Therefore, by setting $|G_{i,2}|=\dots=|G_{i,l-1}|=t$ and summing $|G_{i,1}|$ from $1$ to $t$, the semi-autoregressive modeling task

is equivalent to Equation (3). In our experiments, we set $t=2$ and uniformly sampling $|G_{i,1}|$ from $\{1,2,\dots,t\}$ instead. Since there exist multiple prediction targets for one condition sequence, it is difficult for vanilla Transformer (Vaswani et al., 2017) to model. Therefore, for language tasks, we will leverage and extend the two-stream attention module proposed in XLNet (Yang et al., 2019) by designing causal masks for semi-autoregressive modeling, where the two-stream attention can be used to model more various dependency relationship between tokens compared to vanilla Transformer. More details on the realization will be illustrated in Appendix B.3. Similarly, for vision tasks, we use the two-stream attention version of ViT (Dosovitskiy et al., 2021), which is also adopted in the previous work (Hua et al., 2022).

Language Tasks. The model is pretrained on the Pile dataset (Gao et al., 2020), which contains content from 22 diverse sources. We use a decoder-only Transformer with 16 layers and hidden size of 768. We train the model for 100K steps with batch size of 8192. The other pretraining procedure follows the protocol proposed by Cramming (Geiping & Goldstein, 2023). The model is then finetuned with 5 epochs and evaluated on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018a), which is a collection of several language understanding tasks.

Table 2 presents the GLUE test results. These results reveal that the model trained with the diversity-enhanced autoregressive objective consistently exhibits improvement in classification tasks. Our proposed objective has a gain of 1.9% on the average score and 3.3% on the hardest task CoLA. These improvements support the effectiveness of our theoretical findings regarding the classification ability of autoregressive models.

Vision Tasks. The model is pretrained on ImageNet-1K with ViT-Base and ImageNet-100 with ViT-Small (Deng et al., 2009). The training epoch is 200 with a warm-up of 10 epochs. The batch size is set to 4096 following previous works (He et al., 2022; Hua et al., 2022; Xie et al., 2022). We set the base learning rate as 5e-4 and use Adam optimizer (Kingma & Ba, 2015). After the pretraining is finished, we perform downstream tasks under two protocols: For linear probing, we train a linear classifier on the frozen pretrained encoder for 90 epochs; For non-linear fine-tuning, we train both the pretrained encoder and the linear classifier with the soft target cross entropy loss (Peterson et al., 2019) for 100 epochs. The batch size is set to 4096.

The experiment results are presented in Table 3. The diversity-enhanced autoregressive objective improves linear probing accuracy by $\sim$3%. It also improves fine-tuning accuracy on ViT-S by 2.1% and on ViT-B by 0.4%. These positive outcomes affirm the effectiveness of our proposed objective, emphasizing our theoretical insight that it is beneficial to have more connections in generative SSL.

We also conduct various extensive experiments on vision tasks including few-shot learning as Dubois et al. (2023), multi-epoch training as Li et al. (2023) and transfer learning as Kong et al. (2023).

Few-shot learning. We conduct experiments with the diversity-enhanced autoregressive objective. In the stage of the downstream tasks, we only use 1% of the label. Finetuning and linear probing results are shown in Table 4. As shown in the above table, the diversity-enhanced autoregressive objective improves both the few-shot linear and finetuning accuracy, which further verifies the effectiveness of our proposed objectives across different tasks.

Multi-epoch training. We conduct experiments with the diversity-enhanced autoregressive objective. We respectively pretrain the models with 100 epochs and 200 epochs. Finetuning and linear probing results are shown in Table 5. As shown in the above table, the diversity-enhanced autoregressive models with different training epochs consistently achieve significant improvement in classification tasks, which also verifies the effectiveness of our proposed objectives.

Transfer learning. We conduct experiments with the diversity-enhanced autoregressive objective. We evaluate the transfer learning performance pretrained on ImageNet-1K on 9 downstream datasets as Zhao et al. (2022) and Ericsson et al. (2021), which are FGVC Aircraft, Caltech-101, Stanford Cars, CIFAR10, CIFAR100, DTD, Oxford 102 Flowers, Food-101 and Oxford-IIIT Pets. For linear evaluation, multinomial logistic regression is fit on the extracted features. Results are shown in Table 6. As shown in the table, the diversity-enhanced autoregressive models also show superior performance in transfer learning, which further verifies the generalization performance of our models.

The experiments across three tasks and multiple datasets show that the modified objectives consistently improve the performance of downstream tasks. This proves the superiority of our objective and verifies the effectiveness of our proposed theory.

In this part, we will consider the variable-length MIM objective as proposed in Equation (5). Recall that the variable-length MIM objective leverages variable mask ratio to mitigate the length misalignment problem in masked models. We will consider different sampling strategies for the mask ratio in language tasks and vision tasks.

Language Tasks. Similar to the experiments on diversity-enhanced autoregressive Objective, the model is pretrained on the Pile dataset (Gao et al., 2020). We use a decoder-only Transformer (without causal mask) with 16 layers and hidden size of 768. The pretraining procedure follows the protocol proposed by Cramming (Geiping & Goldstein, 2023). Regarding the mask ratio in the masked model, we explore three options: (1) retaining the original BERT value of 0.15, (2) opting for a larger ratio of 0.3, and (3) uniformly sampling from the range $[0.15,0.3]$, aligning with our proposed variable-length objective. We adopt perplexity as the evaluation metric, which is the exponential of Equation (2). We will evaluate the model on two datasets: (1) the tail of the Pile dataset, which is never exhibited to the model during the pretraining process; (2) a subset of C4 dataset (Raffel et al., 2020), which is a colossal, cleaned version of Common Crawl’s web crawl corpus.

Table 8 demonstrates that the model trained with the variable-length objective exhibits significantly lower perplexity compared to the masked models trained with a fixed mask ratio on both datasets. This suggests that the variable-length objective can improve the generation ability of the model. These enhancements support our theoretical insights into the generation ability of masked models.

Vision Tasks. We select MAE (He et al., 2022) as the baseline model, with pretraining hyper-parameters consistent with those in the diversity-enhanced experiments. After the pretraining is finished, we perform image reconstruction tasks. The image is first randomly masked out with 25% portion. Then the model predicts the masked part token by token in an autoregressive way. Reconstruction quality is evaluated using two metrics between the reconstructed and original images: L2 loss at the pixel level and LPIPS loss (Zhang et al., 2018) at the high-level feature level. The average loss is calculated over the ImageNet validation set.

Table 8 reveals that the model trained with the variable-length objective experiences significantly lower (approximately 30%) reconstruction loss compared to the masked models trained with a fixed mask ratio. These enhancements underscore our theoretical insight that mitigating misalignment of unmasked lengths between pretraining and test examples is crucial for generation ability.