The Power of Contrast for Feature Learning:
A Theoretical Analysis

The idea of contrastive learning was firstly proposed in Hadsell et al. (2006) as an effective method to perform dimensional reduction. Following this line of research, Dosovitskiy et al. (2014) proposed to perform instance discrimination by creating surrogate classes for each instance and Wu et al. (2018) further proposed to preserve a memory bank as a dictionary of negative samples. Other extensions based on this memory bank approach include He et al. (2020); Misra and Maaten (2020); Tian et al. (2020); Chen et al. (2020c). Rather than keeping a costly memory bank, another line of work exploits the benefit of mini-batch training where different samples are treated as negative to each other (Ye et al., 2019; Chen et al., 2020a). Moreover, Khosla et al. (2020) explores the supervised version of contrastive learning where pairs are generated based on label information.

Despite its success in practice, the theoretical understanding of contrastive learning is still limited. Previous works provide provable guarantees for contrastive learning under conditional independence assumption (or its variants) (Arora et al., 2019; Lee et al., 2021; Tosh et al., 2021; Tsai et al., 2020). Specifically, they assume the two contrastive views are independent conditioned on the label and show that contrastive learning can provably learn representations beneficial for in-domain downstream tasks. In addition to this line of research, there exist several alternative perspectives for studying the theoretical properties of contrastive learning. To name a few, Wang and Isola (2020); Graf et al. (2021) explored the representation geometry, HaoChen et al. (2021) analyzed the augmentation graph, Tian (2022) proposed a two-player game theory framework, Zimmermann et al. (2021) demonstrated the connection between contrastive learning and nonlinear Independent Component Analysis (Hyvärinen et al., 2009), Saunshi et al. (2022) showed that the importance of inductive bias in contrastive learning, and Jing et al. (2021) investigated the dimensional collapse phenomenon. Furthermore, Tian et al. (2021); Wang et al. (2021) have also explored the ability of self-supervised learning to learn features even without contrastive pairs, specifically in the context of linear representation settings.

More relevant to this paper, Wen and Li (2021) considered representation learning under the sparse coding model and studied the optimization properties in shallow ReLU neural networks. However, the assumptions that features are extremely sparse and signals follow Gaussian distribution seem strong for real data. Garg and Liang (2020) studied the combination of supervised learning and self-supervised learning. They derived sample complexity bounds in a PAC-learning style for various settings. Specifically, the authors assume that there is a ground-truth representation such that it can keep both self-supervised loss and supervised loss at a very low threshold. However, as the authors admit, it is hard to determine such a threshold in practical settings. For example, since the unlabeled data and labeled data come from different domains, such as Image-Net and CIFAR-10, domain-specific features may have a much lower loss compared with domain-transferable features.

While the aforementioned previous works aim to demonstrate that contrastive learning is capable of learning meaningful representations, it was left untouched why contrastive learning outperforms other representation learning methods. We also shed light on the impact of labeled data in a contrastive learning framework, which is underexplored in prior works. A detailed comparison with existing literature is deferred to Appendix A.1.

This paper is organized as follows. Section 2 provides the setup for the data-generating process and the loss function. In Section 3, we review the connection between PCA and autoencoders/GANs. We also establish a theoretical framework to study contrastive learning in the linear representation setting. Under this framework, we evaluate the feature recovery performance and in-domain downstream task performance of contrastive learning and autoencoders. In Section 4, we analyze the supervised contrastive learning. In Section 5, we verify our theoretical results given in Sections 3 and 4. Finally, we summarize our analysis and provide future directions in Section 6.

In this paper, we use $O,\Omega,\Theta$ to hide universal constants and we write $a_{k}\lesssim b_{k}$ for two sequences of positive numbers $\{a_{k}\}$ and $\{b_{k}\}$ if and only if there exists a universal constant $C>0$ such that $a_{k}<Cb_{k}$ for any $k$. We write $a_{k}\asymp b_{k}$ when $a_{k}\lesssim b_{k}$ and $a_{k}\gtrsim b_{k}$ holds simultaneously. We use $\|\cdot\|,\|\cdot\|_{2},\|\cdot\|_{F}$ to represent the $\ell_{2}$ norm of vectors, the spectral norm of matrices, and Frobenius norm of matrices respectively. Let $\mathbb{O}_{d,r}$ be a set of $d\times r$ orthogonal matrices. Namely, $\mathbb{O}_{d,r}\triangleq\{U\in\mathbb{R}^{d\times r}:U^{\top}U=I_{r}\}$. We write $n\gg d$ when there exists a sufficiently small constant $c$ depending on the constant and independent of $n$, $d$ and $r$ such that $d/n<c$ holds. $d\gg r$ is defined similarly. We use $|A|$ to denote the cardinality of a set $A$. For any $n\in\mathbb{N}^{+}$, let $[n]=\{1,2,\cdots,n\}$. We use $\|\sin\Theta(U_{1},U_{2})\|_{F}$ to refer to the sine distance between two orthogonal matrices $U_{1},U_{2}\in\mathbb{O}_{d,r}$, which is defined by: $\left\|\sin\Theta\left(U_{1},U_{2}\right)\right\|_{F}\triangleq\left\|U_{1\perp}^{\top}U_{2}\right\|_{F}$, where $U_{1\perp}\in\mathbb{O}_{d-r,r}$ is any orthogonal complement of $U_{1}$. More properties of sine distance can be found in Section A.3. We use $\{e_{i}\}_{i=1}^{d}$ to denote the canonical basis in $d$-dimensional Euclidean space $\mathbb{R}^{d}$, that is, $e_{i}$ is the vector whose $i$-th coordinate is $1$ and all the other coordinates are $0$. Let $\mathbb{I}\{A\}$ be an indicator function that takes $1$ when $A$ is true, otherwise takes $0$. We write $a\vee b$ and $a\wedge b$ to denote $\max(a,b)$ and $\min(a,b)$, respectively.

Given an input $x\in\mathbb{R}^{d}$, contrastive learning aims to learn a low-dimensional representation $h=f(x;\theta)\in\mathbb{R}^{r}$ by contrasting different samples, that is, maximizing the agreement between positive pairs, and minimizing the agreement between negative pairs. Suppose we have $n$ data points $X=[x_{1},x_{2},\cdots,x_{n}]\in\mathbb{R}^{d\times n}$ from the population distribution $\mathcal{D}$. The contrastive learning task can be formulated to the following optimization problem:

$$\min_{\theta}\mathcal{L}(\theta)=\min_{\theta}\frac{1}{n}\sum_{i=1}^{n}\ell(x_{i},\mathcal{B}_{i}^{Pos},\mathcal{B}_{i}^{Neg};f(\cdot;\theta))+\lambda R(\theta),$$

(1)

where $\ell(\cdot)$ is a contrastive loss and $\lambda R(\theta)$ is a regularization term; $\mathcal{B}_{i}^{Pos},\mathcal{B}_{i}^{Neg}$ are the sets of positive samples and negative samples corresponding to $x_{i}$, the details of which are described below.

There are two common approaches to generating positive and negative pairs, depending on whether or not label information is available. When the label information is not available, the typical strategy is to generate different views of the original data via augmentation (Hadsell et al., 2006; Chen et al., 2020a). Two views of the same data point serve as the positive pair for each other, while those of different data serve as negative pairs.

The loss function of the self-supervised contrastive learning problem can then be written as:

$$\mathcal{L}_{\text{SelfCon}}(W)\!=-\!\frac{1}{2n}\sum_{i=1}^{n}\sum_{v=1}^{2}\biggl{[}\langle Wg_{v}(x_{i}),Wg_{[2]\setminus\{v\}}(x_{i})\rangle\!-\!\sum_{j\neq i}\sum_{s=1}^{2}\frac{\langle Wg_{v}(x_{i}),Wg_{s}(x_{j})\rangle}{2n-2}\biggr{]}\!+\!\frac{\lambda}{2}\|WW^{\top}\|_{F}^{2}.$$

(3)

In particular, we adopt the following augmentation in our analysis.

When the label information is available, Khosla et al. (2020) proposed the following approach to generate positive and negative pairs.

Correspondingly, the loss function of the supervised contrastive learning problem can be written as:

$$\mathcal{L}_{\text{SupCon}}(W)=-\frac{1}{nK}\sum_{k=1}^{K}\sum_{i=1}^{n}\biggl{[}\sum_{j\neq i}\frac{\langle Wx_{i}^{k},Wx_{j}^{k}\rangle}{n-1}-\sum_{j=1}^{n}\sum_{s\neq k}\frac{\langle Wx_{i}^{k},Wx_{j}^{s}\rangle}{n(K-1)}\biggr{]}+\frac{\lambda}{2}\|WW^{\top}\|_{F}^{2}.$$

(4)

In real-world scenarios, data often comprises both signal (relevant information) and noise (irrelevant distractions). For instance, in image classification, the signal might be the primary subject of interest, while the noise could represent background elements. Self-supervised learning methods, without predefined tasks, aim to extract generalized patterns from data, ideally capturing as much of the signal as possible. It is commonly understood that signals tend to exhibit specific low-complexity structures, often being low-rank and showing higher correlations across coordinates. In contrast, background noise might lack a distinct structure, potentially being dense (or full rank) with lower coordinate correlations. To delve into this structural difference more rigorously, we consider an additive data-generating model. Here, the observed data emerges as a combination of a low-rank signal and dense noise.

$$x=U^{\star}z+\xi,\quad\mathrm{Cov}(z)=\nu^{2}I_{r},\quad\mathrm{Cov}(\xi)=\Sigma,$$

(5)

where $z\in\mathbb{R}^{r}$ and $\xi\in\mathbb{R}^{d}$ are both zero mean sub-Gaussian independent random variables, and $\nu\in\mathbb{R}$ is a constant represents the signal strength. In particular, $U^{\star}\in\mathbb{O}_{d,r}$ and $\Sigma=\operatorname{diag}(\sigma_{1}^{2},\cdots,\sigma_{d}^{2})$. The first term $U^{\star}z$ represents the signal of interest residing in a low-dimensional subspace spanned by the columns of $U^{\star}$. The second term $\xi$ is the dense noise with heteroskedastic noise. Given that, the ideal low-dimensional representation is to compress the observed $x$ into a low-dimensional representation spanned by the columns of $U^{\star}$. This model is known as the spiked covariance model (Johnstone, 2001; Bai and Yao, 2012; Yao et al., 2015; Zhang et al., 2018). It was proposed from the empirical observation that the eigenvalues of the sample covariance matrix of phoneme data have few ”spikes”, which corresponds to the low-dimensional structure of data generation. The model has been used in the literature of PCA (Johnstone, 2001; Deshpande and Montanari, 2014; Zhang et al., 2018) and Contrastive Learning (Wen and Li, 2021).

In this paper, we aim to learn a good projection $W\in\mathbb{R}^{r\times d}$ onto a lower-dimensional subspace from the observation $x$. Since the information of $W$ is invariant with the transformation $W\leftarrow OW$ for any $O\in\mathbb{O}_{r,r}$, the essential information of $W$ is contained in the right eigenvector of $W$. Thus, we quantify the goodness of the representation $W$ using the sine distance $\|\sin\Theta(U,U^{\star})\|_{F}$, where $U$ is the top-$r$ right eigenspace of $W$. It is notable that we only assume that noise and signal follow a sub-Gaussian distribution. This includes bounded noise/signals such as images, sound data, or text data.

Autoencoders are popular unsupervised learning methods to perform dimensional reduction. Autoencoders learn two functions: encoder $f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{r}$ and decoder $g:\mathbb{R}^{r}\rightarrow\mathbb{R}^{d}$. While the encoder $f$ compresses the original data into low-dimensional features, and the decoder $g$ recovers the original data from those features. It can be formulated to be the following optimization problem for samples $\{x_{i}\}_{i=1}^{n}$ (Ballard, 1987; Fan et al., 2019):

$$\min_{f,g}\mathbb{E}_{x}\mathcal{L}(x,g(f(x))).$$

(6)

By minimizing this loss, autoencoders try to preserve the essential features to recover the original data in the low-dimensional representation. In our setting, we consider the class of linear functions for $f$ and $g$. The loss function is set as the mean squared error. Write $f(x)=W_{\text{AE}}x$ and $g(x)=W_{\text{DE}}x$. Namely, we consider the following problem.

$$\min_{W_{\text{AE}},W_{\text{DE}}}\frac{1}{n}\|X-W_{\text{DE}}W_{\text{AE}}X\|_{F}^{2}.$$

Let $X=(x_{1},\dots,x_{n})\in\mathbb{R}^{d\times n}$. By Theorem 2.4.8 in Golub and Loan (1996), the optimal solution is given by the eigenspace of $XX^{\top}$, which exactly corresponds to the result of PCA. Thus, in linear representation settings, autoencoders are equivalent to PCA, which is also often known as undercomplete linear autoencoders (Bourlard and Kamp, 1988; Plaut, 2018; Fan et al., 2019). We write the obtained low-rank representation by autoencoders as

$$W_{\text{AE}}=(U_{\text{AE}}\Sigma_{\text{AE}}V_{\text{AE}}^{\top})^{\top},$$

(7)

where $U_{\text{AE}}$ is the top-$r$ eigenvectors of matrix $XX^{\top}$, $\Sigma_{\text{AE}}$ is a diagonal matrix of spectral values and $V_{\text{AE}}=[v_{1},\cdots,v_{r}]\in\mathbb{R}^{r\times r}$ can be any orthonormal matrix.

We also note that GANs (Goodfellow et al., 2014) is related to PCA. Namely, Feizi et al. (2020) showed that the global solution for GANs recovers the empirical PCA solution as the generative model.

To see this, let $\mathcal{W}_{2}$ be the second-order Wasserstein distance. Also let $\mathcal{G}$ be the set of linear generator functions from $\mathbb{R}^{r}\to\mathbb{R}^{d}$. Consider the following $\mathcal{W}_{2}$ GAN optimization problem:

$\displaystyle\min_{g\in\mathcal{G}}\mathcal{W}_{2}^{2}(\mathbb{P}_{n},\mathbb{P}_{g(Z)}),$

(8)

where $\mathbb{P}_{n}$ denotes the empirical distribution of i.i.d. data $x_{1},\dots,x_{n}\in\mathbb{R}^{d}$ and $\mathbb{P}_{g(Z)}$ is the generated distribution with generator $g$ and $Z\sim N(0,I_{r})$. Note that the optimization problem Equation (8) can be written as $\min_{\mathbb{P}_{n,Z}}\min_{g\in\mathcal{G}}\mathbb{E}[\|X-g(Z)\|^{2}]$, where the first minimization is over probability distributions which have marginals $\mathbb{P}_{n}$ and $\mathbb{P}_{Z}$. By Theorem 2 in Feizi et al. (2020), the optimizer of problem Equation (8) is obtained as $\hat{g}:Z\mapsto\hat{G}Z$, where $\hat{G}$ satisfies $\hat{G}\hat{G}^{\top}=U_{\text{AE}}\Sigma_{\text{AE}}^{2}U_{\text{AE}}^{\top}$. This implies $W_{\text{AE}}^{\top}:\mathbb{R}^{r}\to\mathbb{R}^{d}$ is also a solution to the optimization problem Equation (8). Hence GANs learn the PCA solution as a generator.

From this equivalence among ordinary PCA, autoencoders, and GANs, we only focus on autoencoders hereafter for brevity.

Here we bridge PCA and contrastive learning with certain augmentations under the linear representation setting. Recall that the optimization problem for self-supervised contrastive learning is formulated as:

$$\min_{W\in\mathbb{R}^{r\times d}}\mathcal{L}_{\text{SelfCon}}(W)\!:=-\!\frac{1}{2n}\sum_{i=1}^{n}\sum_{v=1}^{2}\biggl{[}\langle Wg_{v}(x_{i}),Wg_{[2]\setminus\{v\}}(x_{i})\rangle\!-\!\sum_{j\neq i}\sum_{s=1}^{2}\frac{\langle Wg_{v}(x_{i}),Wg_{s}(x_{j})\rangle}{2n-2}\biggr{]}\!+\!\frac{\lambda}{2}\|WW^{\top}\|_{F}^{2}.$$

(9)

To compare contrastive learning with autoencoders, we now derive the solution of the optimization problem (9). We start with the general result for self-supervised contrastive learning with augmented pairs generation in Definition 2.1, and then turn to the special case of random masking augmentation (Definition 2.2).

The proof is given in Appendix B.1.

Proposition 3.1 is a general result for augmented pairs generation with fixed and deterministic augmentation functions. The result itself only depends on the augmented data matrices, thus it is straightforward to generalize to the case where different augmentation functions are applied to different samples, we omit it here for the simplicity of notations. Moreover, when the augmentation is sampled from a stochastic distribution, we can also characterize the optimal solution of the expected loss in the same way. Specifically, if we apply the random masking augmentation (2.2), we can further obtain a result to characterize the optimal solution. For any square matrix $A\in\mathbb{R}^{d\times d}$, we denote $D(A)$ to be $A$ with all off-diagonal entries set to be zero and $\Delta(A)=A-D(A)$ to be $A$ with all diagonal entries set to be zero. Then we have the following corollary for random masking augmentation.

The proof is given in Appendix B.2.

With Proposition 3.1 and Corollary 3.2 established, we can find that the self-supervised contrastive learning equipped with augmented pairs generation and random masking augmentation can eliminate the effect of random noise on the diagonal entries of the observed covariance matrix. Since $\mathrm{Cov}(\xi)=\Sigma$ is a diagonal matrix, when the diagonal entries $\mathrm{Cov}(U^{\star}z)=\nu^{2}U^{\star}U^{\star\top}$ only take a small proportion of the total Frobenius norm, the contrasting augmented pairs will preserve the core features while eliminating most of the random noise and give a more accurate estimation of core features.

After bridging both autoencoder and contrastive learning with PCA, now we can perform the analysis of feature recovery ability to understand the benefit of contrastive learning over autoencoders. As mentioned above, our target is to recover the subspace spanned by the columns of $U^{\star}$, which can further help us obtain information on the unobserved $z$ that is important for in-domain downstream tasks. However, the observed data has a covariance matrix of $\nu^{2}U^{\star}U^{\star\top}+\Sigma$ rather than the desired $\nu^{2}U^{\star}U^{\star\top}$, which brings difficulty to representation learning. We demonstrate that contrastive learning can better exploit the structure of core features and obtain better estimation than autoencoders in this setting.

We start with autoencoders. In the noiseless case, the covariance matrix is $\nu^{2}U^{\star}U^{\star\top}$ and autoencoders can perfectly recover the core features. However, in noisy cases, the random noises sometimes perturb the core features, which makes autoencoders fail to learn the core features. Such noisy cases are widespread in real applications such as measurement errors and backgrounds in images such as grasses and sky. Interestingly, we will later show that contrastive learning can better recover $U^{\star}$ despite the presence of large noise.

To provide rigorous analysis, we first introduce the incoherent constant (Candès and Recht, 2009).

Intuitively, the incoherent constant measures the degree of the incoherence of the distribution of entries among different coordinates, or loosely speaking, the similarity between $U$ and canonical basis $\{e_{i}\}_{i=1}^{d}$. For uncorrelated random noise, the covariance matrix is diagonal and its eigenspace is exactly spanned by the canonical basis $\{e_{i}\}_{i=1}^{d}$ (if the diagonal entries in $\Sigma$ are all different), which attains the maximum value of the incoherent constant. On the contrary, the core features usually exhibit certain correlation structures and the corresponding eigenspace of the covariance matrix is expected to have a lower incoherent constant.

We then introduce a few assumptions which our theoretical results are built on. Recall that in the spiked covariance model (5), $x=U^{\star}z+\xi$, $\mathrm{Cov}(z)=\nu^{2}I_{r}$ and $\mathrm{Cov}(\xi)=\operatorname{diag}(\sigma_{1}^{2},\cdots,\sigma_{d}^{2})$.

The incoherent constant often appears in the literature of matrix completion (Candès and Recht, 2009) and PCA (Zhang et al., 2018). The order of $I(U^{\star})$ can be arbitrary as long as it decreases to $0$ as $d\rightarrow\infty$. One can directly adapt the later results to this setting. If $U$ is distributed uniformly on $\mathbb{O}_{d,r}$, then the expectation of incoherent constant is of order $r\log d/d$.

Thus, we set $I(U^{\star})$ to the order $r\log d/d$ for simplicity. The proof is given in Appendix B.6. Here we provide a remark on the implication of our assumptions, and we defer a further discussion on how to generalize our main results under weaker assumptions in Remark 3.11.

Now we are ready to present our first result, showing that the autoencoders are unable to recover the core features in the large-noise regime. Due to the equivalence among PCA, autoencoders, and GANs we presented in Section 3.1, for brevity, we only focus on autoencoders hereafter.

The proof is given in Appendix B.7. The condition $d\gg r$ means that there exists a sufficiently small constant $c>0$ independent of $d$ and $r$ such that $r/d<c$ holds. The additional assumptions $\{\sigma_{i}^{2}\}_{i=1}^{d}$ are different from each other and $\sigma_{(1)}^{2}/(\sigma_{(r)}^{2}-\sigma_{(r+1)}^{2})<C_{\sigma}$ for some universal constant $C_{\sigma}$ are made to ensure the identifiability of top-$r$ eigenspace. We need these conditions to guarantee the uniqueness of $U_{\text{AE}}$. As an extreme example, the top-$r$ eigenspace of the identity matrix can be any $r$-dimensional subspace and thus not unique. To avoid discussing such arbitrariness of the output, we make these assumptions to guarantee the separability of the eigenspace.

Then we investigate the feature recovery ability of the self-supervised contrastive learning approach.

The proof is given in Appendix B.9. The two terms in equation (15) can be explained as follows: the first term is due to the shift between the distributions of the augmented data and the original data. Specifically, the random masking augmentation generates two views with disjoint nonzero coordinates and thus can mitigate the influence of random noise on the diagonal entries in the covariance matrix. However, such augmentation slightly hurts the estimation of core features. This bias, appearing as the first term in Equation (15), is measured by the incoherent constant defined in Equation (12). The second term corresponds to the estimation error of the population covariance matrix.

Theorems 3.9 and 3.10 characterize the difference in feature recovery ability between autoencoders and contrastive learning. The autoencoders fail to recover most of the core features in the large-noise regime since $\|\sin\Theta(U,U^{\star})\|_{F}$ has a trivial upper bound $\sqrt{r}$. In contrast, with the help of data augmentation, the contrastive learning approach mitigates the corruption of random noise while preserving core features. As $n$ and $d$ increase, it yields a consistent estimator of core features and further leads to better performance in the in-domain downstream tasks, as shown in the next section.

In the previous section, we have seen that contrastive learning can recover the core feature effectively. In practice, we are interested in using the learned features on in-domain downstream tasks. He et al. (2020) experimentally showed the overwhelming performance of linear classifiers trained on representations learned with contrastive learning against several supervised learning methods in those in-domain downstream tasks.

Following the recent success, here we evaluate the in-domain downstream performance of simple predictors, which take a linear transformation of the representation as an input. Let $W_{\text{CL}}$ and $W_{\text{AE}}$ be the learned representations based on train data $X\in\mathbb{R}^{n\times d}$. We observe a new signal $\check{x}=U^{\star}\check{z}+\check{\xi}$ independent of $X$ following the spiked covariance model (5). For simplicity, assume $\check{z}$ follows $N(0,\nu^{2}I_{r})$ and is independent of $\check{\xi}$. We consider two major types of in-domain downstream tasks: classification and regression. For the binary classification task, we observe a new supervised sample $\check{y}$ following the binary response model:

$\displaystyle\check{y}|\check{z}$

$\displaystyle\sim\text{Ber}(F(\langle\check{z},w^{\star}\rangle/\nu)),$

(18)

where $F:\mathbb{R}\to[0,1]$ is a known monotone increasing function satisfying $1-F(u)=F(-u)$ for any $u\in\mathbb{R}$ , and $w^{\star}\in\mathbb{R}^{r}$ is a unit vector of coefficients. Notice that our model (18) includes a logistic model (when $F(u)=1/(1+e^{-u})$) and probit models (when $F(u)=\Phi(u)$, where $\Phi$ is the cumulative distribution function of the standard normal distribution.) We can also interpret model (18) as a shallow neural network model with width $r$ for binary classification. For the regression task, we observe a new supervised sample $\check{y}$ following the linear regression model:

$\displaystyle\check{y}$

$\displaystyle=\langle\check{z},w^{\star}\rangle/\nu+\check{\epsilon},$

(19)

where $\check{\epsilon}\sim(0,\sigma_{\epsilon}^{2})$ is independent of $\check{z}$, , and $w^{\star}\in\mathbb{R}^{r}$ is a unit vector of coefficients as before. We can interpret this model as a principal component regression model (PCR) (Jolliffe, 1982) under standard error-in-variables settings²²2 In error-in-variables settings, the bias term of the measurement error appears in prediction and estimation risk. Since our focus lies in proving a better performance of contrastive learning against autoencoders, we ignore the unavoidable bias term here by considering the excess risk. , where we assume that the coefficients lie in a low-dimensional subspace spanned by column vectors of $U^{\star}$. We either estimate or predict the signal based on the observed samples contaminated by the measurement error $\check{\xi}$. For details of PCR in error-in-variables settings, see, for example, Ćevid et al. (2020); Agarwal et al. (2020); Bing et al. (2021).

In classification setting, we specify $0$-$1$ loss, that is, $\ell_{c}(\delta)\triangleq\mathbb{I}\{\check{y}\neq\delta(\check{x})\}$ for some predictor $\delta$ taking values in $\{0,1\}$. For regression task, we employ the squared error loss $\ell_{r}(\delta)\triangleq(\check{y}-\delta(\check{x}))^{2}$. Based on some learned representation $W$, we consider a class of linear predictors. Namely, $\delta_{W,w}(\check{x})\triangleq\mathbb{I}\{F(w^{\top}W\check{x})\geq 1/2\}$ for classification task and $\delta_{W,w}(\check{x})\triangleq w^{\top}W\check{x}$ for regression task, where $w\in\mathbb{R}^{r}$ is a weight vector $w\in\mathbb{R}^{r}$. Note that the learned representation depends only on unsupervised samples $X$. Let $\mathbb{E}_{\mathcal{D}}[\cdot]$ and $\mathbb{E}_{\mathcal{E}}[\cdot]$ the expectations with respect to $(X,Z)$ and $(\check{y},\check{x},\check{z})$, respectively.

Our goal as stated above is to bound the prediction risk of predictors $\{\delta_{W,w}:w\in\mathbb{R}^{r}\}$ constructed upon the learned representations $W_{\text{CL}}$ and $W_{\text{AE}}$, that is, the quantity $\inf_{w\in\mathbb{R}^{r}}\mathbb{E}_{\mathcal{E}}[\ell(\delta_{W_{\text{CL}},w})]$ and $\inf_{w\in\mathbb{R}^{r}}\mathbb{E}_{\mathcal{E}}[\ell(\delta_{W_{\text{AE}},w})]$.

Now we state our results on the performance of the in-domain downstream prediction task.

The proofs are given in Appendix B.11.

This result shows that the price of estimating $U^{\star}$ by contrastive learning on an in-domain downstream prediction task can be made small in the case where the core feature lies in a relatively low-dimensional subspace, and the number of samples is relatively large compared to the ostensible dimension of data.

However, the in-domain downstream performance of autoencoders is not as good as contrastive learning. We obtain the following lower bound for the in-domain downstream prediction risk with the autoencoders.

The proof is given in Appendix B.11. The condition $n\gg d\gg r$ means that there exists a sufficiently small constant $c>0$ independent of $n$, $d$ and $r$ such that $d/n\vee r/d<c$ holds.

The constant $c^{\prime}$ appearing in Theorem 3.14 is a constant term independent of $d$ and $n$. Thus, when $d$ is sufficiently large compared to $r$ and $d/n$ is small, the upper bound of in-domain downstream task performance via contrastive learning in Theorem 3.13 is smaller than the lower bound of in-domain downstream task performance via autoencoders. The assumption of $r\leq r_{c}$ in Theorem 3.14 is assumed for clarity of presentation. Using the same techniques in the proof of Theorem 3.14, one can obtain a constant lower bound for autoencoders with slightly stronger assumptions, for example, $\rho^{2}=O(1/\log d)$ with $n\gg dr$, without assuming $r\leq r_{c}$. Our theory can be adapted to both of these assumptions. Our results support the empirical success of contrastive learning.

We first demonstrate the impact of labels in contrastive learning under the standard single-sourced (i.e. no transfer learning) setting. Suppose our samples are drawn from $r+1$ different classes with probability $p_{k}$ for class $k\in[r+1]$, and $\sum_{k=1}^{r+1}p_{k}=1$. For each class, samples are generated from a class-specific Gaussian distribution:

$$x^{k}=\mu^{k}+\xi^{k},\quad\xi^{k}\sim\mathcal{N}(0,\Sigma^{k}),\quad\forall k=1,2,\cdots,r+1.$$

(20)

We assume the norms of $\mu^{k},\forall k\in[r+1]$ are in the same order, that is, denote $\nu=\|\mu^{1}\|/\sqrt{r}$, we have $\|\mu^{k}\|=O(\sqrt{r}\nu),\forall k\in[r+1]$. We further assume $\Sigma^{k}=\operatorname{diag}(\sigma_{1,k}^{2},\cdots,\sigma_{d,k}^{2})$, denote $\sigma_{(1)}^{2}=\max_{1\leq i\leq d,1\leq j\leq r+1}\sigma_{i,j}^{2}$ and assume $\sum_{k=1}^{r+1}p_{k}\mu^{k}=0$, where the last assumption is added to ensure identifiability since the classification problem (20) is invariant under translation. Denote $\Lambda=\sum_{k=1}^{r+1}p_{k}\mu^{k}\mu^{k\top}$, we assume $\operatorname{rank}(\Lambda)=r$ and $C_{1}\nu^{2}<\lambda_{(r)}(\Lambda)<\lambda_{(1)}(\Lambda)<C_{2}\nu^{2}$ for two universal constants $C_{1}$ and $C_{2}$. We remark that this model is a labeled version of the spiked covariance model (5) since the core features and random noise are both sub-Gaussian. We use $r+1$ classes to ensure that $\mu^{k}$’s span an $r$-dimensional space, and denote its orthonormal basis as $U^{\star}$. Recall that our target is to recover $U^{\star}$.

As introduced in Definition 2.4, the supervised contrastive learning introduced by Khosla et al. (2020) allows us to generate contrastive pairs using labeled information and discriminate instances across classes. When we have both labeled data and unlabeled data, we can perform contrastive learning based on pairs that are generated separately for the two types of data.

In this section, we show that the theoretical tools developed in this paper can be used to illustrate the role of label information when using contrastive learning in the transfer learning setting. Label information can tell us the beneficial information for the in-domain downstream task, and learning with labeled data will filter out useless information and preserve the decisive parts of core features. However, in transfer learning, the label information is sometimes rather found to hurt the performance of contrastive learning. For example, in Table 4 of Islam et al. (2021), while supervised contrastive learning gains 8% improvement in source tasks by incorporating label information, it improves only 1% on generalizing to new datasets on average and can even hurt on some datasets. Such observation implies that label information in contrastive learning has very different roles for generalization on source tasks and new tasks. In this section, we consider two regimes of transfer learning – tasks are insufficient/abundant. In both regimes, we provide theories to support the empirical observations and further demonstrate how to wisely combine supervised and self-supervised contrastive learning to avoid those harms and achieve better performance. Specifically, we consider a transfer learning problem with regression setting and binary classification setting. Suppose we have $T$ source tasks which share a common data generative model (5). For the $t$-th task, the labels are generated by $y^{t}=\langle w_{t},z\rangle/\nu$ in a regression setting while $y^{t}=\operatorname{sign}(\langle w_{t},z\rangle)$ in a binary classification setting, where $w_{t}\in\mathbb{R}^{r}$ is a unit vector varying across tasks. These two settings share the same distribution for $x$ only differ in the way to generate the labels $y^{t}$.

To incorporate label information, we maximize the Hilbert-Schmidt Independence Criteria (HSIC) (Gretton et al., 2005; Barshan et al., 2011), which has been widely used in literature (Song et al., 2007a, b, c; Barshan et al., 2011).