Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds and Benign Overfitting

The traditional understanding of machine learning suggests that models with zero training error tend to overfit, and explicit regularization is often necessary to achieve good generalization. Given the empirical success of deep learning models with zero training error [58, 55] and the (re-)discovery of the “double descent” phenomenon [61], however, it has become clear that the textbook U-shaped learning curve is only part of a larger picture: it is possible for an overparameterized model with zero training loss to achieve low population error in a noisy setting. In an effort to understand how interpolation learning occurs, there has been much recent study of the testbed problem of linear regression with Gaussian features [[, e.g.]]bartlett2020benign, tsigler2020benign, BHX:two-models, hastie:surprises, JLL:basis-pursuit, junk-feats, muthukumar:interpolation, negrea:in-defense. Significant progress has been made in this setting, including nearly-matching necessary and sufficient conditions for consistency of the minimal $\ell_{2}$ norm interpolator [66].

Despite the fundamental role of uniform convergence in statistical learning theory, most of this line of work has used other techniques to analyze the particular minimal-norm interpolator.¹¹1[71] argue that [66]’s proof technique is fundamentally based on uniform convergence of a surrogate predictor; [76] study a closely related setting with a uniform convergence-type argument, but do not establish consistency. We discuss both papers in more detail in Section 4. Instead of directly analyzing the population error of a learning algorithm, a uniform convergence-type argument would control the worst-case generalization gap over a class of predictors containing the typical outputs of a learning rule. Typically, this is done because for many algorithms – unlike the minimal Euclidean norm interpolator – it is difficult to exactly characterize the learned predictor, but we may be able to say e.g. that its norm is not too large. Since uniform convergence does not tightly depend on a specific algorithm, the resulting analysis can highlight the key properties that lead to good generalization: it can give bounds not only for, say, the minimal-norm interpolator, but also for other interpolators with low norm [[, e.g.]]junk-feats, increasing our confidence that low norm – and not some other property the particular minimal-norm interpolator happens to have – is key to generalization. In linear regression, practical training algorithms may not always find the exact minimal Euclidean norm solution, so it is also reassuring that all interpolators with sufficiently low Euclidean norm generalize.

[64], however, raised significant questions about the applicability of typical uniform convergence arguments to certain high-dimensional regimes, similar to those seen in interpolation learning. Following their work, [73, 65, 76, 71] all demonstrated the failure of forms of uniform convergence in various interpolation learning setups. To sidestep these negative results, [73] suggested considering bounds which are uniform only over predictors with zero training error. This weaker notion of uniform convergence has been standard in analyses of “realizable” (noiseless) learning at least since the work of [39, Chapter 6.4] and [40]. [73] demonstrated that at least in one particular noisy setting, such uniform convergence is sufficient for showing consistency of the minimal $\ell_{2}$ norm interpolator, even though “non-realizable” uniform convergence arguments (those over predictors regardless of their training error) cannot succeed. It remains unknown, however, whether these types of arguments can apply to more general linear regression problems and more typical asymptotic regimes, particularly showing rates of convergence rather than just consistency.

In this work, we show for the first time that uniform convergence is indeed able to explain benign overfitting in general high-dimensional Gaussian linear regression problems. Similarly to how the standard analysis for learning with Lipschitz losses bounds generalization gaps through Rademacher complexity [[, e.g.]]understanding-ML, our Theorem 1 (Section 3) establishes a finite-sample high probability bound on the uniform convergence of the error of interpolating predictors in a hypothesis class, in terms of its Gaussian width. This is done through an application of the Gaussian Minimax Theorem; see the proof sketch in Section 7. Combined with an analysis of the norm of the minimal $\ell_{2}$ norm interpolator (Theorem 2 in Section 4), our bound recovers known consistency results [66], as well as proving a conjectured upper bound for larger-norm interpolators [73].

In addition, since we do not restrict ourselves to Euclidean norm balls but instead consider interpolators in an arbitrary compact set, our results allows for a wide range of other applications. Our analysis leads to a natural extension of the consistency result and notions of effective rank of [66] for arbitrary norms (Theorem 5 in Section 5). As a demonstration of our general theory, in Section 6 we show novel consistency results for the minimal $\ell_{1}$ norm interpolator (basis pursuit) in particular settings, which we believe are the first results of their kind.

To state our results, we first need to introduce some key tools.

The radius measures the size of a set in the Euclidean norm. The Gaussian width of a set $S$ can be interpreted as the number of dimensions that a random projection needs to approximately preserve the norms of points in $S$ [57, 42]. These two complexity measures are connected by Gaussian concentration: Gaussian width is the expected value of the supremum of some Gaussian process, and the radius upper bounds the typical deviation of that supremum from its expected value.

Note that $\Sigma=\Sigma_{1}\oplus\Sigma_{2}$ effectively splits the eigenvectors of $\Sigma$ into two disjoint parts.

We can now state our generic bound. Section 7 sketches the proof; all full proofs are in the appendix.

In our applications, we consider $\mathcal{K}=\{w\in\mathbb{R}^{d}:\|w\|\leq B\}$ for an arbitrary norm, with $B$ based on a high-probability upper bound for $\left\lVert\hat{w}\right\rVert$. Depending on the application, the rank of $\Sigma_{1}$ will be either constant or $o(n)$, so that $\beta\to 0$. The term $\left\lVert w^{*}\right\rVert_{\Sigma_{2}}$ generally does not scale with $n$ and hence is often negligible. As hinted earlier, we can think of the Gaussian width term and the radius term as bias and variance, respectively. To achieve consistency, we can expect that the Gaussian width should scale as $\sigma\sqrt{n}$. This agrees with the intuition that we need increasing norm to memorize noise when the model is not realizable. The radius term requires some care in our applications, but can be handled by the covariance splitting technique. As part of the analysis in the following sections, we will rigorously show in many settings that the dominant term in the upper bound is the Gaussian width. In these cases, our upper bound is roughly $W(\Sigma_{2}^{1/2}\mathcal{K})^{2}/n$, which can be viewed as the ratio between the (probabilistic) dimension of our hypothesis class and sample size. We will also analyze how large $\mathcal{K}$ must be to contain any interpolators, allowing us to find consistency results.

It can be easily seen that the Gaussian width of a Euclidean norm ball reduces nicely to the product of the norm of our predictor with the typical norm of $x$: if $\mathcal{K}=\{w\in\mathbb{R}^{d}:\|w\|_{2}\leq B\}$, then

Therefore, it is plausible that ($\star$ ‣ 2) holds with $\psi_{n}=\operatorname*{\mathbb{E}}\|x\|_{2}^{2}=\operatorname{Tr}(\Sigma)$. Figure 1 illustrates this generalization bound in two simple examples, motivated by [63, 73]. Indeed, an application of our main theorem proves that this is exactly the case for Gaussian data.

The above bound is clean and simple, but only proves a sub-optimal rate of $n^{-1/4}$. This is because the choice of covariance split used in the proof of Corollary 1 uses no information about the particular structure of $\Sigma$. This bound can also be slightly loose in situations where the eigenvalues of $\Sigma$ decay rapidly, in which case $\operatorname{Tr}(\Sigma)$ can be replaced by a smaller quantity. We next state a more precise bound on the generalization error, which requires introducing the following notions of effective rank.

The $r(\Sigma)$ rank can roughly be understood as the squared ratio between the Gaussian width and radius in our previous bound. It is related to the concentration of the $\ell_{2}$ norm of a Gaussian vector with covariance $\Sigma$. In fact, both definitions of effective ranks can be derived by applying Bernstein’s inequality to ${\|x\|^{2}}/{\operatorname*{\mathbb{E}}\|x\|^{2}}$. We will only need $r(\Sigma)$ in the generalization bound below, but we will show in Theorem 2 that $R(\Sigma)$ can be used to control the norm of the minimal norm interpolator $\hat{w}$.

In order to use Corollary 1 or 2 to prove consistency, we need a high-probability bound for $\left\lVert\hat{w}\right\rVert_{2}$, the norm of the minimal norm interpolator, so that $B$ will be large enough to contain any interpolators. Theorem 2 gives exactly such a bound, showing that if the effective ranks $R(\Sigma_{2})$ and $r(\Sigma_{2})$ are large, then we can construct an interpolator with Euclidean norm nearly $\|w^{*}\|_{2}+\sigma\sqrt{n/\operatorname{Tr}(\Sigma_{2})}$.

Plugging in estimates of $\|\hat{w}\|$ to our scale-sensitive bound Corollary 2, we obtain a population loss guarantee for $\hat{w}$ in terms of effective ranks.

From (7), we can see that to ensure consistency, i.e. $L(\hat{w})\to\sigma^{2}$, it is enough that $\gamma\to 0$, $\epsilon\to 0$, and $\|w^{*}\|_{2}\sqrt{\frac{\operatorname{Tr}(\Sigma_{2})}{n}}\to 0$. Recalling the definitions of the various quantities and using that $r(\Sigma_{2})^{2}\geq R(\Sigma_{2})$ [66, Lemma 5], we arrive at the following conditions.

All the results on the Euclidean setting are special cases of the following results for arbitrary norms. It is worth keeping in mind that the Euclidean norm will still play a role in these analyses, via the Gaussian width and radius appearing in Theorem 1 and the $\ell_{2}$ projection $P$ appearing in Theorem 4.

The Euclidean norm’s dual is itself; for it and many other norms, $\partial\left\lVert u\right\rVert_{*}$ is a singleton set. Using these notions, we will now give versions of the effective ranks appropriate for generic norm balls.

The first effective $\left\lVert\cdot\right\rVert$-rank is the squared ratio of Gaussian width to the radius of the set $\Sigma^{1/2}\mathcal{K}$, where $\mathcal{K}$ is a norm ball $\{w:\left\lVert w\right\rVert\leq B\}$; the importance of this ratio should be clear from Theorem 1. The Gaussian width is given by $W(\Sigma^{\frac{1}{2}}\mathcal{K})=B\operatorname*{\mathbb{E}}\left\lVert x\right\rVert_{*}$, while the radius can be written $\sup_{\left\lVert w\right\rVert\leq B}\left\lVert w\right\rVert_{\Sigma}$ so that the factors of $B$ cancel.

The choice of $R_{\left\lVert\cdot\right\rVert}$ arises naturally from our bound on $\left\lVert\hat{w}\right\rVert$ in Theorem 4 below. Large effective rank of $\Sigma_{2}$ means the sub-gradient $v^{*}$ of $\|\Sigma_{2}^{1/2}H\|_{*}$ is small in the $\lVert\cdot\rVert_{\Sigma_{2}}$ norm. This is, in fact, closely related to the existence of low-norm interpolators. First, note that $\Sigma_{2}^{1/2}H$ corresponds to the small-eigenvalue components of the covariate vector. For $v^{*}$ to be a sub-gradient means that moving the weight vector $w$ in the direction of $v^{*}$ is very effective at changing the prediction $\langle w,X\rangle$; having small $\lVert\cdot\rVert_{\Sigma_{2}}$ norm means that moving in this direction has a very small effect on the population loss $L(w)$. Together, this means the sub-gradient will be a good direction for benignly overfitting the noise.

Using the general notion of effective ranks, we can find an analogue of Corollary 2 for general norms.

As in the Euclidean special case, we still need a bound on the norm of $\hat{w}=\operatorname*{arg\,min}_{\hat{L}(w)=0}\left\lVert w\right\rVert$ to use this result to study the consistency of $\hat{w}$. This leads us to the second main technical result of this paper, which essentially says that if the effective ranks $R_{\left\lVert\cdot\right\rVert}(\Sigma_{2})$ and $r_{\left\lVert\cdot\right\rVert}(\Sigma_{2})$ are sufficiently large, then there exists an interpolator with norm $\left\lVert w^{*}\right\rVert+\sigma\sqrt{n}/\operatorname*{\mathbb{E}}\lVert\Sigma_{2}^{1/2}g\rVert_{*}$.

For a specific choice of norm $\left\lVert\cdot\right\rVert$, we can verify that $\|v^{*}\|_{\Sigma_{2}}$ is small. In the Euclidean case, for example, this is done by (78) in Section C.2; in our basis pursuit application to come, this is done by (92). The term $\epsilon_{2}$ measures the cost of using a projected version of the subgradient; in most of our applications, we can take $\epsilon_{2}=0$. Recalling that $\lVert v^{*}\rVert=1$, this is obviously true with the Euclidean norm for any $\Sigma_{2}$. More generally, if $\Sigma$ is diagonal, then it is natural to only consider covariance splits $\Sigma=\Sigma_{1}\oplus\Sigma_{2}$ such that $\Sigma_{2}$ is diagonal. Then, when $\left\lVert\cdot\right\rVert$ is the $\ell_{1}$ norm (or $\ell_{p}$ norms more generally), it can be easily seen that $Pv^{*}=v^{*}$ and so $\|Pv^{*}\|=\|v^{*}\|=1$.

Straightforwardly combining Corollaries 3 and 4 yields the following theorem, which gives guarantees for minimal-norm interpolators in terms of effective rank conditions. Just as in the Euclidean case, we can extract from this result a simple set of sufficient conditions for consistency of the minimal norm interpolator.

The theory for the minimal $\ell_{1}$ norm interpolator, $\hat{w}_{\mathit{BP}}\in\arg\min_{\hat{L}(w)=0}\left\lVert w\right\rVert_{1}$ – also known as basis pursuit [44] – is much less developed than that of the minimal $\ell_{2}$ norm interpolator. In this section, we illustrate the consequences of our general theory for basis pursuit. Full statements and proofs of results in this section are given in Appendix E.

The dual of the $\ell_{1}$ norm is the $\ell_{\infty}$ norm $\left\lVert u\right\rVert_{\infty}=\max_{i}\lvert u_{i}\rvert$, and $\partial\|u\|_{\infty}$ is the convex hull of $\{\operatorname{sign}(u_{i})\,e_{i}:i\in\arg\max|u_{i}|\}$. From the definition of sub-gradient, we observe that

Furthermore, by convexity we have

and so $r_{\|\cdot\|_{1}}(\Sigma)=\frac{\left(\operatorname*{\mathbb{E}}\|\Sigma^{1/2}g\|_{\infty}\right)^{2}}{\max_{i}\Sigma_{ii}}\leq R_{\|\cdot\|_{1}}(\Sigma)$. Therefore, we can use a single notion of effective rank. For simplicity, we denote $r_{1}(\Sigma)=r_{\|\cdot\|_{1}}(\Sigma)$. Combining this with (13) and the previous discussion of (14), we obtain the following sufficient conditions for consistency of basis pursuit.

A key ingredient in our analysis is a celebrated result from Gaussian process theory known as the Gaussian Minmax Theorem (GMT) [41, 56]. Since the seminal work of [45], the GMT has seen numerous applications to problems in statistics, machine learning, and signal processing [[, e.g.]]stojnic2013framework,deng2019model,oymak2010new,oymak2018universality. Most relevant to us is the work of [56], which introduced the Convex Gaussian Minmax Theorem (CGMT) and developed a framework for the precise analysis of regularized linear regression. Here we apply the GMT/CGMT to study uniform convergence and the norm of the minimal norm interpolator.

In this work, we prove a generic generalization bound in terms of the Gaussian width and radius of a hypothesis class. We also provide a general high probability upper bound for the norm of the minimal norm interpolator. Combining these results, we recover the sufficient conditions from [66] in the $\ell_{2}$ case, confirm the conjecture of [73] for Gaussian data, and obtain novel consistency results in the $\ell_{1}$ case. Our results provide concrete evidence that uniform convergence is indeed sufficient to explain interpolation learning, at least in some settings.

A future direction of our work is to extend the main results to settings with non-Gaussian features; this has been achieved in other applications of the GMT [59], and indeed we expect that a version of ($\star$ ‣ 2) likely holds for non-Gaussian data as well. Another interesting problem is to study uniform convergence of low-norm near-interpolators, and characterize the worst-case population error as the norm and training error both grow. This could lead to a more precise understanding of early stopping, by connecting the optimization path with the regularization path. Finally, it is unknown whether our sufficient conditions for consistency in Section 5 are necessary, and it remains a challenge to apply uniform convergence of interpolators to more complex models such as deep neural networks.

Frederic Koehler was supported in part by E. Mossel’s Vannevar Bush Faculty Fellowship ONR-N00014-20-1-2826. Research supported in part by NSF IIS award 1764032, NSF HDR TRIPODS award 1934843, and the Canada CIFAR AI Chairs program. This work was done as part of the Collaboration on the Theoretical Foundations of Deep Learning (deepfoundations.ai).

[sorting=nyt]