Generalization error of min-norm interpolators in transfer learning

We consider the canonical transfer learning setting where random samples are observed from two different populations. Formally, we observe two datasets $(\bm{y}^{(1)},\bm{X}^{(1)})$ and $(\bm{y}^{(2)},\bm{X}^{(2)})$—one serves as the source data and the other the target. Typically, the former has more sample size than the latter. (Some of our results extend to the general case of multiple source data, see Section 3.4). We assume that the observed data comes from linear models given by

$$\bm{y}^{(k)}=\bm{X}^{(k)}\bm{\beta}^{(k)}+\bm{\epsilon}^{(k)},k=1,2,$$

(2.1)

where $k=1$ and $k=2$ corresponds to the source and target respectively. Here, $\mathbf{y}^{(k)}\in\mathbb{R}^{n_{k}}$ denotes the response vectors, with $n_{k}$ representing the number of samples observed in population, $\mathbf{X}^{(k)}\in\mathbb{R}^{n_{k}\times p}$ denotes the observed covariate matrix from population $k$ with $p$ the number of features, $\bm{\beta}^{(k)}\in\mathbb{R}^{p}$ are signal vectors independent of $\bm{X}^{(k)}$, and $\bm{\epsilon}^{(k)}\in\mathbb{R}^{n_{k}}$ are noise vectors independent of both $\mathbf{X}^{(k)}$ and $\bm{\beta}^{(k)}$. We denote $n=n_{1}+n_{2}$ as the total number of samples.

Throughout the paper, we make the following assumptions on $\bm{X}^{(k)}$ and $\bm{\epsilon}^{(k)}$ which are standard in the random matrix literature [55, 2]. First, for the covariates $\bm{X}^{(k)}$, we assume

$$\bm{X}^{(k)}=\bm{Z}^{(k)}(\bm{\Sigma}^{(k)})^{1/2},$$

(2.2)

where $\bm{Z}^{(k)}\in\mathbb{R}^{n_{k}\times p}$ are random matrices whose entries $Z^{(k)}_{ij}$ are independent random variables with zero mean and unit variance. We further assume that there exists a constant $\tau>0$ by which the $\varphi$-th moment of each entry is bounded for some $\varphi>4$:

$$\mathbb{E}\left[\left|Z_{ij}^{(k)}\right|^{\varphi}\right]\leq\tau^{-1}.$$

(2.3)

The eigenvalues of $\bm{\Sigma}^{(k)}$, denoted as $\lambda^{(k)}_{1},\ldots,\lambda^{(k)}_{p}$ are bounded:

$$\tau\leq\lambda_{p}^{(k)}\leq...\leq\lambda_{1}^{(k)}\leq\tau^{-1}.$$

(2.4)

Second, the noises $\bm{\epsilon}^{(k)}$ are assumed to have i.i.d. entries with mean $0$, variance $\sigma^{2}$, and bounded moments up to any order. That is, for any $\phi>0$, there exists a constant $C_{\phi}$ such that

$$\mathbb{E}\left[\left|\epsilon_{i}^{(k)}\right|^{\phi}\right]\leq C_{\phi}.$$

(2.5)

Lastly, as our analysis pertains to the regime where sample size is smaller than feature dimensions, and the ratio between sample size and feature dimensions remain bounded. Denoting $\gamma_{1}:=p/n_{1},\gamma_{2}:=p/n_{2},\gamma:=p/n$, we assume

$$1+\tau\leq\gamma_{1},\gamma_{2},\gamma\leq\tau^{-1}.$$

(2.6)

As displayed, we are particularly interested in the overparameterized regime where $p>n$, partly because it is more prevalent in the modern machine learning applications. Besides, the underparamterized regime, i.e. $p<n$, is better understood in literature, cf. [57] (they assumed $n_{2}>p$, but their results remain valid for the general case $n>p$).

For convenience we summarize all our assumptions below:

We remark that we do not require any additional assumptions on the signals $\bm{\beta}^{(k)}$, except for their independence on $\bm{X}^{(k)}$. In fact, our results later reveal that the convergences of risks of interest is conditional on $\bm{\beta}^{(k)}$, and only depends on $\bm{\beta}^{(k)}$ through their norms.

Data fusion, the integration of information from diverse datasets, has been broadly categorized into three approaches in literature: early, intermediate and late fusion [3, 49]. Early fusion combines information from multiple sources at the input level. Late fusion learns independently from each dataset and combines information at the end. Intermediate fusion achieves a middle ground and attempts to jointly learn information from both datasets in a more sophisticated manner than simply concatenating them. In this work, we consider a form of min-norm interpolation that captures both early and a class of intermediate fusions strategies. To achieve early fusion using min-norm interpolation, one would naturally consider the pooled min-$\ell_{2}$-norm interpolator, defined as

$$\hat{\bm{\beta}}=\operatorname*{arg\,min}\left\{\|\bm{b}\|_{2}\ \ \text{s.t.}\ \bm{y}^{(k)}=\bm{X}^{(k)}\bm{b},\ k=1,2\right\}.$$

(2.7)

This interpolator admits the following analytical expression:

$$\hat{\bm{\beta}}=\left(\bm{X}^{(1)\top}\bm{X}^{(1)}+\bm{X}^{(2)\top}\bm{X}^{(2)}\right)^{\dagger}\left(\bm{X}^{(1)\top}\bm{y}^{(1)}+\bm{X}^{(2)\top}\bm{y}^{(2)}\right),$$

(2.8)

where for any matrix $\bm{A}$, we denote its pseudoinverse by $\bm{A}^{\dagger}$. This interpolator is alternatively designated as the ”ridgeless” estimator, owing to it being the limit of the ridge estimator when the penalty term vanishes (noticed earlier by [29] in the context of a single training data):

	$\displaystyle\hat{\bm{\beta}}$	$\displaystyle=\lim_{\lambda\rightarrow 0}\left(\bm{X}^{(1)\top}\bm{X}^{(1)}+\bm{X}^{(2)\top}\bm{X}^{(2)}+n\lambda\bm{I}\right)^{-1}\left(\bm{X}^{(1)\top}\bm{y}^{(1)}+\bm{X}^{(2)\top}\bm{y}^{(2)}\right)$		(2.9)
		$\displaystyle=\lim_{\lambda\rightarrow 0}\operatorname*{arg\,min}_{\bm{b}}\left\{\frac{1}{n}\|\bm{y}^{(1)}-\bm{X}^{(1)}\bm{b}\|_{2}^{2}+\frac{1}{n}\|\bm{y}^{(2)}-\bm{X}^{(2)}\bm{b}\|_{2}^{2}+\lambda\|\bm{b}\|_{2}^{2}\right\}.$

Coincidentally, although the pooled min-$\ell_{2}$-norm intepolator is motivated via early fusion, it can also capture a form of intermediate fusion as described below. If we consider a weighted ridge loss with weights $w_{1},w_{2}$ on the source and target data respectively, then the ridgeless limit of this loss recovers the pooled min-$\ell_{2}$-norm interpolator. This can be seen as follows: for any two weighting coefficient $w_{1},w_{2}>0$,

	$\displaystyle\hat{\bm{\beta}}$	$\displaystyle=\operatorname*{arg\,min}\left\{\|\bm{b}\|_{2}\ \ \text{s.t.}\ w_{k}\bm{y}^{(k)}=w_{k}\bm{X}^{(k)}\bm{b},\ k=1,2\right\}$		(2.10)
		$\displaystyle=\left(w_{1}\bm{X}^{(1)\top}\bm{X}^{(1)}+w_{2}\bm{X}^{(2)\top}\bm{X}^{(2)}\right)^{\dagger}\left(w_{1}\bm{X}^{(1)\top}\bm{y}^{(1)}+w_{2}\bm{X}^{(2)\top}\bm{y}^{(2)}\right)$
		$\displaystyle=\lim_{\lambda\rightarrow 0}\left(w_{1}\bm{X}^{(1)\top}\bm{X}^{(1)}+w_{2}\bm{X}^{(2)\top}\bm{X}^{(2)}+n\lambda\bm{I}\right)^{-1}\left(w_{1}\bm{X}^{(1)\top}\bm{y}^{(1)}+w_{2}\bm{X}^{(2)\top}\bm{y}^{(2)}\right)$
		$\displaystyle=\lim_{\lambda\rightarrow 0}\operatorname*{arg\,min}_{\bm{b}}\left\{\frac{w_{1}}{n}\|\bm{y}^{(1)}-\bm{X}^{(1)}\bm{b}\|_{2}^{2}+\frac{w_{2}}{n}\|\bm{y}^{(2)}-\bm{X}^{(2)}\bm{b}\|_{2}^{2}+\lambda\|\bm{b}\|_{2}^{2}\right\},$

indicating that the interpolator above uniquely stands as the common solution of both the min-$\ell_{2}$-norm interpolator and the ridgeless estimator when two datasets are weighted by $w_{1},w_{2}$ in the intermediate fusion loss formulation.

Furthermore, the pooled min-$\ell_{2}$-norm interpolator (2.7) appears naturally as the limit of the gradient descent based linear estimator in the following way: Initialize $\bm{\beta}_{0}=0$ and run gradient descent with iterates given by

$$\bm{\beta}_{t+1}=\bm{\beta}_{t}+\eta\sum_{k=1}^{2}\bm{X}^{(k)\top}(\bm{y}^{(k)}-\bm{X}^{(k)}\bm{\beta}_{t}),\qquad t=0,1,\ldots,$$

with $0<\eta\leq\min\{\lambda_{\max}(\bm{X}^{(1)\top}\bm{X}^{(1)}),\lambda_{\max}(\bm{X}^{(2)\top}\bm{X}^{(2)})\}$, then $\lim_{t\rightarrow\infty}\bm{\beta}_{t}=\hat{\bm{\beta}}$.

In recent years, min-norm interpolated type solutions and their statistical properties have been extensively studied in numerous contexts [5, 8, 10, 11, 15, 37, 39, 38]. In fact, it is conjectured that this min-norm regularization is responsible for superior statistical behavior of estimators in overparametrized regime. In light of these results and the aforementioned abundant formulations, we view our work as a first step of understanding the behavior of this important family of estimators in transfer learning, under the lens of early and intermediate fusion.

We conclude this section by highlighting that computing such an interpolator requires only the summary statistics $\mathbf{X}^{(k)\top}\mathbf{X}^{(k)}$ and $\mathbf{X}^{(k)\top}\mathbf{y}^{(k)}$ for $k=1,2$, rather than entire datasets. This characteristic is particularly beneficial in fields such as medical or genomics research, where individual-level data may be highly sensitive and not permissible for sharing across institutions.

To assess the performance of $\hat{\bm{\beta}}$, we focus on its prediction risk at a new (unseen) test point $(\bm{y}_{0},\bm{x}_{0})$ drawn from the target distribution, i.e., $k=2$. This out-of-sample prediction risk (hereafter simply referred to as risk) is defined as

$$R(\hat{\bm{\beta}};\bm{\beta}^{(2)}):=\mathbb{E}\left[(\bm{x}^{\top}_{0}\hat{\bm{\beta}}-\bm{x}^{\top}_{0}\bm{\beta}^{(2)})^{2}\hskip 2.15277pt|\hskip 2.15277pt\bm{X}\right]=\mathbb{E}\left[\|\hat{\bm{\beta}}-\bm{\beta}^{(2)}\|_{\bm{\Sigma}^{(2)}}^{2}\hskip 2.15277pt|\hskip 2.15277pt\bm{X}\right],$$

(2.11)

where $\|\bm{x}\|_{\bm{\Sigma}}^{2}:=\bm{x}^{\top}\bm{\Sigma}\bm{x}$ and $\bm{X}:=(\bm{X}^{(1)\top},\bm{X}^{(2)\top})^{\top}$. Note that this risk has a $\sigma^{2}$ difference from the mean-squared prediction error for the new data point, which does not affect the relative performance and is therefore omitted. Also, this risk definition takes expectation over both the randomness in the new test point $(\bm{y}_{0},\bm{x}_{0})$ and the randomness in the training noises $(\bm{\epsilon}^{(1)},\bm{\epsilon}^{(2)})$. The risk is conditional on the covariate $\bm{X}$. Despite this dependence, we will use the notation $R(\hat{\bm{\beta}},\bm{\beta}^{(2)})$ for conciseness since the context is clear. The risk admits a bias-variance decomposition:

$$R(\hat{\bm{\beta}};\bm{\beta}^{(2)})=\underbrace{\|\mathbb{E}(\hat{\bm{\beta}}|\bm{X})-\bm{\beta}^{(2)}\|_{\bm{\Sigma}^{(2)}}^{2}}_{B(\hat{\bm{\beta}};\bm{\beta}^{(2)})}+\underbrace{\textnormal{Tr}[\textnormal{Cov}(\hat{\bm{\beta}}|\bm{X})\bm{\Sigma}^{(2)}]}_{V(\hat{\bm{\beta}};\bm{\beta}^{(2)})}.$$

(2.12)

Plugging in (2.9), we immediately obtain the following result which we will crucially use for the remainder of the sequel:

In the next two sections, we will layout our main results which involve precise risk characterizations of the pooled min-$\ell_{2}$-norm interpolator. Bias and variance terms, as decomposed in Lemma 2.1, will be presented separately.

Throughout the rest of the paper, we denote $f(p)=O(g(p))$ if there exists a constant $C$ such that $|f(p)|\leq Cg(p)$ for large enough $p$. What’s more, we say an event $\mathcal{E}$ happens with high probability (w.h.p.) if $\mathbb{P}(\mathcal{E})\rightarrow 1$ as $p\rightarrow\infty$.

Recall the formula for the risk $R(\hat{\bm{\beta}},\bm{\beta}^{(2)})$ in (2.12) and its decomposition in Lemma 2.1. In this sequel, we make a further isotropic assumption that $\bm{\Sigma}^{(1)}=\bm{\Sigma}^{(2)}=\bm{I}$ and leave generalizations for future works. We will split the risk into bias and variance following Lemma 2.1, and characterize their behavior respectively. Note $\bm{\Sigma}^{(1)}=\bm{\Sigma}^{(2)}=\bm{I}$ yields $B_{3}(\hat{\bm{\beta}};\bm{\beta}^{(2)})=0$. Therefore, our contribution focuses on characterizing the asymptotic behavior of $B_{1}(\hat{\bm{\beta}};\bm{\beta}^{(2)}),B_{2}(\hat{\bm{\beta}};\bm{\beta}^{(2)})$ and $V(\hat{\bm{\beta}};\bm{\beta}^{(2)})$ as defined by (2.13) and (2.14). Our main result in this regard is given below.

We remark that Theorem 3.1 holds true for finite values of $n$ and $p$. It characterizes the bias and variance of the pooled min-$\ell_{2}$-norm interpolator for the overparametrized regime $p>n$. In fact, it shows that the bias can be decomposed into two terms (3.2) and (3.3), which depend only on the $\ell_{2}$-norm of $\bm{\beta}^{(2)}$ and $\tilde{\bm{\beta}}$ respectively. Therefore, the generalization error is independent of the angle between $\bm{\beta}^{(1)}$ and $\bm{\beta}^{(2)}$. The variance $V(\hat{\bm{\beta}};\bm{\beta}^{(2)})$ is decreasing in $p/n$, but the bias $B(\hat{\bm{\beta}};\bm{\beta}^{(2)})$ is not necessarily monotone. We provide extensive numerical examples of $R(\hat{\bm{\beta}};\bm{\beta}^{(2)})$ in Section 3.5 showing that the risk function still exhibits the well-known double-descent phenomena. The main challenge in establishing Theorem 3.1 lies in the analysis of $B_{2}(\hat{\bm{\beta}};\bm{\beta}^{(2)})$. To this end, we develop a novel local law that involves extensive use of free probability. More details and explanations are provided in Section 5. The theorem allows us to answer the following questions:

1. (i)

   When does the pooled min-$\ell_{2}$-norm interpolator outperform the target-based min-$\ell_{2}$-norm interpolator? (Section 3.2)

2. (ii)

   What is the optimal target sample size that minimizes the generalization error of pooled min-$\ell_{2}$-norm interpolator? (Section 3.3)

In this section, we compare the risk of $\hat{\bm{\beta}}$, the pooled min-$\ell_{2}$-norm interpolator, with that of the interpolator using only the target dataset, defined as follows:

$$\hat{\bm{\beta}}^{(2)}=\operatorname*{arg\,min}\|\bm{b}\|_{2}\ \ \text{s.t.}\ \bm{y}^{(2)}=\bm{X}^{(2)}\bm{b}.$$

(3.4)

The risk of $\hat{\beta}^{(2)}$ has been previously studied:

The above proposition, along with our results of model shift (Theorem 3.1 and Proposition 3.2), allow us to characterize when pooled min-$\ell_{2}$-norm interpolator yields lower risk than the target-only min-$\ell_{2}$-norm interpolator, i.e. when data pooling leads to positive/negative transfer:

The implications of Proposition 3.3 are as-follows:

1. (i)

   When the SNR is small, i.e. $\rm{SNR}\leq\frac{1}{(1-1/\gamma)(1-1/\gamma_{2})}$, adding more data always hurts the pooled min-$\ell_{2}$-norm interpolator. Note that, even if the source and target populations have the exact same model settings, pooling the data induces higher risk for the inteprolator.

2. (ii)

   If SNR is large, whether pooling the data helps or not depends on the level of model shift. Namely, if SSR exceeds $\rho$ defined by (3.7), the models are far apart resulting in negative transfer. Otherwise, pooling the datasets help. In particular if $\rm{SSR}\geq\frac{1-1/\gamma}{1-1/\gamma_{1}}$, then for any values of $\rm{SNR}$, data-pooling harms.

3. (iii)

   Since the $\rm{SNR}$ and $\rm{SSR}$ can be consistently estimated using the available samples (see (3.3)), Proposition 3.3 provides a data-driven method for deciding whether to use the pooled interpolator or the target-based interpolator.

In sum, in the over-parametrized regime, positive transfer happens when the model shift is low and the SNR is sufficiently high. Figure 1 illustrates our findings in a simulation setting.

Our findings in the previous section highlights that, to achieve the least possible risk for pooled min-$\ell_{2}$-norm interpolator, one should neither always pool both datasets nor always use all samples in either datasets. In fact, if SNR and SSR were known, it would be possible to identify subsets of source and target datasets, having sample size $\tilde{n}_{1},\tilde{n}_{2}$ respectively, such that the risk of the pooled min-$\ell_{2}$-norm interpolator based on the total data of size $\tilde{n}_{1}+\tilde{n}_{2}$ is minimized. To this end, the purpose of this section is two-fold: first to provide an estimation strategy for SNR and SSR, and second to invoke the estimator and provide a guideline for choosing appropriate sample size.

For SNR and SSR estimation, we employ a simplified version of the recently introduced HEDE method [50]. Let $\hat{\bm{\beta}}_{d}^{(k)}$ denote the following debiased Lasso estimator [32, 12]:

$$\hat{\bm{\beta}}_{d}^{(k)}=\hat{\bm{\beta}}_{\rm{L}}^{(k)}+\frac{\bm{X}^{(k)\top}(\bm{y}^{(k)}-\bm{X}^{(k)}\hat{\bm{\beta}}_{\rm{L}}^{(k)})}{n_{k}-\|\hat{\bm{\beta}}_{\rm{L}}^{(k)}\|_{0}}$$

that is based on the Lasso estimator for some fixed $\lambda_{\rm{L}}$:

$$\hat{\bm{\beta}}_{\rm{L}}^{(k)}:=\operatorname*{arg\,min}_{\bm{b}}\frac{1}{2n}\|\bm{y}^{(k)}-\bm{X}^{(k)}\bm{b}\|_{2}^{2}+\frac{\lambda_{\textrm{L}}}{\sqrt{n_{k}}}\|\bm{b}\|_{1},$$

and let

$$\hat{\tau}_{k}^{2}=\frac{\|\bm{y}^{(k)}-\bm{X}^{(k)}\hat{\bm{\beta}}_{\textrm{L}}^{(k)}\|_{2}^{2}}{(n_{k}-\|\hat{\bm{\beta}}_{\textrm{L}}^{(k)}\|_{0})^{2}}$$

be the corresponding estimated variance for $k=1,2$. Then by Theorem 4.1 and Proposition 4.2 in [50], we know

	$\displaystyle\|\hat{\bm{\beta}}_{d}^{(k)}\|_{2}^{2}-p\hat{\tau}^{2}_{k}\rightarrow\|\bm{\beta}^{(k)}\|_{2}^{2},\ \ k=1,2,$
	$\displaystyle\widehat{\textnormal{Var}}(\bm{y}^{(k)})-(\|\hat{\bm{\beta}}_{d}^{(k)}\|_{2}^{2}-p\hat{\tau}^{2}_{k})\rightarrow\sigma^{2},\ \ k=1,2,$
	$\displaystyle\|\hat{\bm{\beta}}^{d}_{1}-\hat{\bm{\beta}}^{d}_{2}\|_{2}^{2}-p(\hat{\tau}^{2}_{1}+\hat{\tau}^{2}_{2})\rightarrow\|\bm{\beta}^{(1)}-\bm{\beta}^{(2)}\|_{2}^{2}.$

where the last line invokes independence between the datasets. Hence, we obtain the following consistent estimators for SNR and SSR:

	$\displaystyle\widehat{\text{SNR}}:=\frac{\|\hat{\bm{\beta}}_{d}^{(2)}\|_{2}^{2}-p\hat{\tau}^{2}_{2}}{\widehat{\textnormal{Var}}(\bm{y}^{(2)})-(\|\hat{\bm{\beta}}_{d}^{(2)}\|_{2}^{2}-p\hat{\tau}^{2}_{2})}\rightarrow$	$\displaystyle\rm{SNR},$
	$\displaystyle\widehat{\text{SSR}}:=\frac{\|\hat{\bm{\beta}}^{d}_{1}-\hat{\bm{\beta}}^{d}_{2}\|_{2}^{2}-p(\hat{\tau}^{2}_{1}+\hat{\tau}^{2}_{2})}{\|\hat{\bm{\beta}}_{d}^{(2)}\|_{2}^{2}-p\hat{\tau}^{2}_{2}}\rightarrow$	$\displaystyle\rm{SSR}.$		(3.9)

Given the estimators of SNR and SSR, denoted by $\widehat{\text{SNR}}$ and $\widehat{\text{SSR}}$ respectively, an important question arises: what is the optimal size of the target dataset for which the pooled min-$\ell_{2}$-norm interpolator achieves least mean-squared prediction error $R(\hat{\bm{\beta}};\bm{\beta}^{(2)})$? This is answered by the following proposition:

This proposition validates our intuition for the problem: If SNR is low or SSR is large, large amount of target data is not necessary for achieving good generalization. In fact, given the estimators $\widehat{\rm{SNR}}$, $\widehat{\rm{SSR}}$, a data-dependent estimated optimal target data size is given by

$$\hat{n}_{2,\text{opt}}=\left(p-n_{1}-\sqrt{\frac{p^{2}}{\widehat{\rm{SNR}}}+n_{1}\widehat{\rm{SSR}}}\right)_{+}$$

Our results for model shift, i.e., Theorem 3.1 can be extended to the general framework of multiple sources and one target datasets. Denote the target as $T$ and $K$ sources as $1,...,k$, and further denote the signal shift vectors as $\tilde{\bm{\beta}}^{(k)}:=\bm{\beta}^{(k)}-\bm{\beta}^{(T)}$. Assuming we are still in the overparametrized regime $p>n_{1}+...+n_{K}+n_{T}$, and consider the pooled min-$\ell_{2}$-norm interpolator as

$$\hat{\bm{\beta}}=\operatorname*{arg\,min}\|\bm{b}\|_{2}\ \ \text{s.t.}\ \bm{y}^{(T)}=\bm{X}^{(T)}\bm{b},\ \bm{y}^{(k)}=\bm{X}^{(k)}\bm{b},\ k=1,...,K.$$

(3.11)

Then the following theorem charactarizes the bias and variance of the pooled min-$\ell_{2}$-norm interpolator under this setting:

The proof of Proposition 3.5 is similar to that of Theorem 3.1 and is therefore omitted.

In this Section, we demonstrate the applicability of our results for moderate sample sizes, as illustrated in Figure 1. Specifically, we explore the generalization error of the pooled min-$\ell_{2}$-norm interpolator in a scenario where both the source and target data follow isotropic Gaussian distributions. The signals are generated as follows: $\beta_{i}^{(2)}\sim\mathcal{N}(0,\sigma_{\beta}^{2}/p)$ and $\beta_{i}^{(1)}=\beta_{i}^{(2)}+\mathcal{N}(0,\sigma_{s}^{2}/p)$ (and thereafter considered fixed). The signal shift is thus given by $\tilde{\bm{\beta}}\sim\mathcal{N}(0,\sigma_{s}^{2}/p)$. We display the generalization error across varied combinations of source and target sample sizes (denoted by $n_{1}$ and $n_{2}$ respectively), number of covariates ($p$), signal-to-noise ratios (SNRs), and shift-to-signal ratios (SSRs). For completeness, we also present plots of the generalization error for the underparameterized regime, where $p<n$, a regime previously explored in the literature [57].

Figure 1 shows the empirical generalization errors of the pooled min-$\ell_{2}$-norm interpolator closely match the theoretical predictions from Theorem 3.1 in moderate dimensions. Additionally, we present the generalization error of the target-only interpolator, as established by [29]. The risk of the target-only interpolator matches with that of our pooled interpolator when $n_{1}=0$, validating our results.

Considering the OLS estimator’s generalization error for $p<n$, the concatenated error curve reveals a double descent phenomenon. For smaller sample size from the source distribution, the risk of the pooled min-$\ell_{2}$-norm interpolator is a decreasing function of $n_{1}$, rendering the pooled interpolator superior than the target-only interpolator. However, if the SSR value is too high, the first stage of descent is missing, rendering the pooled min-$\ell_{2}$-norm interpolator inferior to its target-only counterpart. Figures 1 (b) and (c) display this dependence on the SNR and SSR values. Namely, higher signal strength and less model shift magnitudes lead to a wider range of $n_{1}$ for which the pooled interpolator is better. Figure 1 (a) further demonstrates the intricate dependence of this dichotomy on $n_{1},n_{2}$ and $p$, indicating that the pooled interpolator performs particularly well when $n_{1},n_{2}$ are much smaller than $p$, which is commonplace in most modern datasets. These findings reinforce and complement the conclusions from Proposition 3.3, providing a more complete picture regarding the positive/negative effects of transfer learning for the pooled min-$\ell_{2}$-interpolator under overparametrization.

We assume that the source and target covariance matrices $(\bm{\Sigma}^{(1)},\bm{\Sigma}^{(2)})$ are simultaneously diagonalizable, as defined below:

The simultaneous diagonalizability assumption has previously been used in the transfer learning literature (cf. [42]). It ensures that the difference in the covariate distributions of the source and the target is solely captured by the eigenvalues $\lambda_{i}^{(1)},\lambda_{i}^{(2)}$. Characterizing the generalization error when covariance matrices lack a common orthonormal basis appears to be fairly technical, and we defer this to future work.

Note that in the typical overparameterized setup with a single dataset, the risk of the min-$\ell_{2}$-norm interpolator is expressed in terms of the empirical distribution of eigenvalues and the angle between the signal and the eigenvector of the covariance matrix (equation (9) in [29]). Consequently, it is natural to expect that the generalization error of our estimator will depend on $\hat{H}_{p}$ and $\hat{G}_{p}$ defined by (4.1), which indeed proves to be the case. The following presents our main result concerning covariate shift.