On the Generalization Power of
Overfitted Two-Layer Neural Tangent Kernel Models

Recently, there is significant interest in understanding why overparameterized deep neural networks (DNNs) can still generalize well (Zhang et al., 2017; Advani et al., 2020), which seems to defy the classical understanding of bias-variance tradeoff in statistical learning (Bishop, 2006; Hastie et al., 2009; Stein, 1956; James & Stein, 1992; LeCun et al., 1991; Tikhonov, 1943). Towards this direction, a recent line of study has focused on overparameterized linear models (Belkin et al., 2018b, 2019; Bartlett et al., 2020; Hastie et al., 2019; Muthukumar et al., 2019; Ju et al., 2020; Mei & Montanari, 2019). For linear models with simple features (e.g., Gaussian features and Fourier features) (Belkin et al., 2018b, 2019; Bartlett et al., 2020; Hastie et al., 2019; Muthukumar et al., 2019; Ju et al., 2020), an interesting “double-descent” phenomenon has been observed. Thus, there is a region where the number of model parameters (or linear features) is larger than the number of samples (and thus overfitting occurs), but the generalization error actually decreases with the number of features. However, linear models with these simple features are still quite different from nonlinear neural networks. Thus, although such results provide some hint why overparameterization and overfitting may be harmless, it is still unclear whether similar conclusions apply to neural networks.

In this paper, we are interested in linear models based on the neural tangent kernel (NTK) (Jacot et al., 2018), which can be viewed as a useful intermediate step towards modeling nonlinear neural networks. Essentially, NTK can be seen as a linear approximation of neural networks when the weights of the neurons do not change much. Indeed, Li & Liang (2018); Du et al. (2018) have shown that, for a wide and fully-connected two-layer neural network, both the neuron weights and their activation patterns do not change much after gradient descent (GD) training with a sufficiently small step size. As a result, such a shallow and wide neural network is approximately linear in the weights when there are a sufficient number of neurons, which suggests the utility of the NTK model.

Despite its linearity, however, characterizing the double descent of such a NTK model remains elusive. The work in Mei & Montanari (2019) also studies the double-descent of a linear version of two-layer neural network. It uses the so-called “random-feature” model, where the bottom-layer weights are random and fixed, and only the top-layer weights are trained. (In comparison, the NTK model for such a two-layer neural network corresponds to training only the bottom-layer weights.) However, the setting there requires the number of neurons, the number of samples, and the data dimension to all grow proportionally to infinity. In contrast, we are interested in the setting where the number of samples is given, and the number of neurons is allowed to be much larger than the number of samples. As a consequence of the different setting, in Mei & Montanari (2019) eventually only linear ground-truth functions can be learned. (Similar settings are also studied in d’Ascoli et al. (2020).) In contrast, we will show that far more complex functions can be learned in our setting. In a related work, Ghorbani et al. (2019) shows that both the random-feature model and the NTK model can approximate highly nonlinear ground-truth functions with a sufficient number of neurons. However, Ghorbani et al. (2019) mainly studies the expressiveness of the models, and therefore does not explain why overfitting solutions can still generalize well. To the best of our knowledge, our work is the first to characterize the double-descent of overfitting solutions based on the NTK model.

Specifically, in this paper we study the generalization error of the min $\ell_{2}$-norm overfitting solution for a linear model based on the NTK of a two-layer neural network with ReLU activation that has no bias. Only the bottom-layer weights are trained. We are interested in min $\ell_{2}$-norm overfitting solutions because gradient descent (GD) can be shown to converge to such solutions while driving the training error to zero (Zhang et al., 2017) (see also Section 2). Given a class of ground truth functions (see details in Section 3), which we refer to as “learnable functions,” our main result (Theorem 1) provides an upper bound on the generalization error of the min $\ell_{2}$-norm overfitting solution for the two-layer NTK model with $n$ samples and $p$ neurons (for any finite $p$ larger than a polynomial function of $n$). This upper bound confirms that the generalization error of the overfitting solution indeed exhibits descent in the overparameterized regime when $p$ increases. Further, our upper bound can also account for the noise in the training samples.

Our results reveal several important insights. First, we find that the (double) descent of the overfitted two-layer NTK model is drastically different from that of linear models with simple Gaussian or Fourier features (Belkin et al., 2018b, 2019; Bartlett et al., 2020; Hastie et al., 2019; Muthukumar et al., 2019). Specifically, for linear models with simple features, when the number of features $p$ increases, the generalization error will eventually grow again and approach the so-called “null risk” (Hastie et al., 2019), which is the error of a trivial model that predicts zero. In contrast, for the class of learnable functions described earlier, the generalization error of the overfitted NTK model will continue to descend as $p$ grows to infinity, and will approach a limiting value that depends on the number of samples $n$. Further, when there is no noise, this limiting value will decrease to zero as the number of samples $n$ increases. This difference is shown in Fig. 1(a). As $p$ increases, the test mean-square-error (MSE) of min-$\ell_{1}$ and min-$\ell_{2}$ overfitting solutions for Fourier features (blue and red curves) eventually grow back to the null risk (the black dashed line), even though they exhibit a descent at smaller $p$. In contrast, the error of the overfitted NTK model continues to descend to a much lower level.

The second important insight is that the aforementioned behavior critically depends on the ground-truth function belonging to the class of “learnable functions.” Further, this class of learnable functions depend on the specific network architecture. For our NTK model (with RELU activation that has no bias), we precisely characterize this class of learnable functions. Specifically, for ground-truth functions that are outside the class of learnable functions, we show a lower bound on the generalization error that does not diminish to zero for any $n$ and $p$ (see Proposition 2 and Section 4). This difference is shown in Fig. 1(b), where we use an almost identical setting as Fig. 1(a), except a different ground-truth function. We can see in Fig. 1(b) that the test-error of the overfitted NTK model is always above the null risk and looks very different from that in Fig. 1(a). We note that whether certain functions are learnable or not critically depends on the specific structure of the NTK model, such as the choice of the activation unit. Recently, (Satpathi & Srikant, 2021) shows that all polynomials can be learned by 2-layer NTK model with ReLU activation that has a bias term, provided that the number of neurons $p$ is sufficiently large. (See further discussions in Remark 2. However, (Satpathi & Srikant, 2021) does not characterize the descent of generalization errors as $p$ increases.) This difference in the class of learnable functions between the two settings (ReLU with or without bias) also turns out to be consistent with the difference in the expressiveness of the neural networks. That is, shallow networks with biased-ReLU are known to be universal function approximators (Ji et al., 2019), while those without bias can only approximate the sum of linear functions and even functions (Ghorbani et al., 2019).

A closely related result to ours is the work in Arora et al. (2019), which characterizes the generalization performance of wide two-layer neural networks whose bottom-layer weights are trained by gradient descent (GD) to overfit the training samples. In particular, our class of learnable functions almost coincides with that of Arora et al. (2019). This is not surprising because, when the number of neurons is large, NTK becomes a close approximation of such two-layer neural networks. In that sense, the results in Arora et al. (2019) are even more faithful in following the GD dynamics of the original two-layer network. However, the advantage of the NTK model is that it is easier to analyze. In particular, the results in this paper can quantify how the generalization error descends with $p$. In contrast, the results in Arora et al. (2019) provide only a generalization bound that is independent of $p$ (provided that $p$ is sufficiently large), but do not quantify the descent behavior as $p$ increases. Our numerical results in Fig. 1(a) suggest that, over a wide range of $p$, the descent behavior of the NTK model (the green curve) matches well with that of two-layer neural networks trained by gradient descent (the cyan curve). Thus, we believe that our results also provide guidance for the latter model. The work in (Fiat et al., 2019) studies a different neural network architecture with gated ReLU, whose NTK model turns out to be the same as ours. However, similar to Arora et al. (2019), the result in Fiat et al. (2019) does not capture the speed of descent with respect to $p$ either. Second, Arora et al. (2019) only provides upper bounds on the generalization error. There is no corresponding lower bound to explain whether ground-truth functions outside a certain class are not learnable. Our result in Proposition 2 provides such a lower bound, and therefore more completely characterizes the class of learnable functions. (See further comparison in Remark 1 of Section 3 and Remark 3 of Section 5.) Another related work Allen-Zhu et al. (2019) also characterizes the class of learnable functions for two-layer and three-layer networks. However, Allen-Zhu et al. (2019) studies a training method that takes a new sample in every iteration, and thus does not overfit all training data. Finally, our paper studies generalization of NTK models for the regression setting, which is different from the classification setting that assumes a separability condition, e.g., in (Ji & Telgarsky, 2019).

We assume the following data model $y=f(\bm{x})+\epsilon$, with the input $\bm{x}\in\mathds{R}^{d}$, the output $y\in\mathds{R}$, the noise $\epsilon\in\mathds{R}$, and $f:\ \mathds{R}^{d}\mapsto\mathds{R}$ denotes the ground-truth function. Let $(\mathbf{X}_{i},\ y_{i})$, $i=1,2,\cdots,n$ denote $n$ training samples. We collect them as $\mathbf{X}=[\mathbf{X}_{1}\ \mathbf{X}_{2}\ \cdots\ \mathbf{X}_{n}]\in\mathds{R}^{d\times n}$, $\bm{y}=[y_{1}\ y_{2}\ \cdots\ y_{n}]^{T}\in\mathds{R}^{n}$, $\bm{\epsilon}=[\epsilon_{1}\ \epsilon_{2}\ \cdots\ \epsilon_{n}]^{T}\in\mathds{R}^{n}$, and $\mathbf{F}(\mathbf{X})=[f(\mathbf{X}_{1})\ f(\mathbf{X}_{2})\ \cdots\ f(\mathbf{X}_{n})]^{T}\in\mathds{R}^{n}$. Then, the training samples can be written as $\bm{y}=\mathbf{F}(\mathbf{X})+\bm{\epsilon}$. After training (to be described below), we denote the trained model by the function $\hat{f}$. Then, for any new test data $\bm{x}$, we will calculate the test error by $|\hat{f}(\bm{x})-f(\bm{x})|$, and the mean squared error (MSE) by $\operatorname*{\mathsf{E}}_{\bm{x}}[\hat{f}(\bm{x})-f(\bm{x})]^{2}$.

For training, consider a fully-connected two-layer neural network with $p$ neurons. Let $\bm{w}_{j}\in\mathds{R}$ and $\mathbf{V}_{0}[j]\in\mathds{R}^{d}$ denote the top-layer and bottom-layer weights, respectively, of the $j$-th neuron, $j=1,2,\cdots,p$ (see Fig. 2). We collect them into $\bm{w}=[\bm{w}_{1}\ \bm{w}_{2}\ \cdots\ \bm{w}_{p}]^{T}\in\mathds{R}^{p}$, and $\mathbf{V}_{0}=[\mathbf{V}_{0}[1]^{T}\ \mathbf{V}_{0}[2]^{T}\ \cdots\ \mathbf{V}_{0}[p]^{T}]^{T}\in\mathds{R}^{dp}$ (a column vector with $dp$ elements). Note that with this notation, for any row or column vector $\bm{v}$ with $dp$ elements, $\bm{v}[j]$ denotes a (row/column) vector that consists of the $(jd+1)$-th to $(jd+d)$-th elements of $\bm{v}$. We choose ReLU as the activation function for all neurons and there is no bias term in the ReLU activation function.

Now we are ready to introduce the NTK model (Jacot et al., 2018). We fix the top-layer weights $\bm{w}$, and let the initial bottom-layer weights $\mathbf{V}_{0}$ be randomly chosen. We then train only the bottom-layer weights. Let $\mathbf{V}_{0}+\overline{\Delta\mathbf{V}}$ denote the bottom-layer weights after training. Thus, the change of the output after training is

In the NTK model, one assumes that $\overline{\Delta\mathbf{V}}$ is very small. As a result, $\bm{1}_{\{\bm{x}^{T}(\mathbf{V}_{0}[j]+\overline{\Delta\mathbf{V}}[j])>0\}}=\bm{1}_{\{\bm{x}^{T}\mathbf{V}_{0}[j]>0\}}$ for most $\bm{x}$. Thus, the change of the output can be approximated by

where $\Delta\mathbf{V}\in\mathds{R}^{dp}$ is given by $\Delta\mathbf{V}[j]\mathrel{\mathop{:}}=\bm{w}_{j}\overline{\Delta\mathbf{V}}[j],\ j=1,2,\cdots,p$, and $\bm{h}_{\mathbf{V}_{0},\bm{x}}\in\mathds{R}^{1\times(dp)}$ is given by

In the NTK model, we assume that the output of the trained model is exactly given by Eq. (1), i.e.,

In other words, the NTK model can be viewed as a linear approximation of the two-layer network when the change of the bottom-layer weights is small.

Define $\mathbf{H}\in\mathds{R}^{n\times(dp)}$ such that its $i$-th row is $\mathbf{H}_{i}\mathrel{\mathop{:}}=\bm{h}_{\mathbf{V}_{0},\mathbf{X}_{i}}$. Throughout the paper, we will focus on the following min-$\ell_{2}$-norm overfitting solution

Whenever $\Delta\mathbf{V}^{\ell_{2}}$ exists, it can be written in closed form as

The reason that we are interested in $\Delta\mathbf{V}^{\ell_{2}}$ is that gradient descent (GD) or stochastic gradient descent (SGD) for the NTK model in Eq. (2) is known to converge to $\Delta\mathbf{V}^{\ell_{2}}$ (proven in Supplementary Material, Appendix B).

Using Eq. (2) and Eq. (3), the trained model is then

In the rest of the paper, we will study the generalization error of Eq. (4).

We collect some assumptions. Define the unit sphere in $\mathds{R}^{d}$ as: $\mathcal{S}^{d-1}\mathrel{\mathop{:}}=\left\{\bm{v}\in\mathds{R}^{d}\ |\ \|\bm{v}\|_{2}=1\right\}$. Let $\mu(\cdot)$ denote the distribution of the input $\bm{x}$. Without loss of generality, we make the following assumptions: (i) the inputs $\bm{x}$ are i.i.d. uniformly distributed in $\mathcal{S}^{d-1}$, and the initial weights $\mathbf{V}_{0}[j]$’s are i.i.d. uniformly distributed in all directions in $\mathds{R}^{d}$; (ii) $p\geq n/d$ and $d\geq 2$; (iii) $\mathbf{X}_{i}\nparallel\mathbf{X}_{j}$ for any $i\neq j$, and $\mathbf{V}_{0}[k]\nparallel\mathbf{V}_{0}[l]$ for any $k\neq l$. We provide detailed justification of those assumptions in Supplementary Material, Appendix C.

We now show that the generalization performance of the overfitted NTK model in Eq. (4) crucially depends on the ground-truth function $f(\cdot)$, where good generalization performance only occurs when the ground-truth function is “learnable.” Below, we first describe a candidate class of ground-truth functions, and explain why they may correspond to the class of “learnable functions.” Then, we will give an upper-bound on the generalization performance for this class of ground-truth functions. Finally, we will give a lower-bound on the generalization performance when the ground-truth functions are outside of this class.

Note that in Definition 1, $\stackrel{{\scriptstyle\text{a.e.}}}{{=}}$ means two functions equals almost everywhere, and $\|g\|_{1}\mathrel{\mathop{:}}=\int_{\mathcal{S}^{d-1}}|g(\bm{z})|d\mu(\bm{z})$. The function $g(\bm{z})$ may be any finite-value function in $L^{1}(\mathcal{S}^{d-1}\mapsto\mathds{R})$. Further, we also allow $g(\bm{z})$ to contain (as components) Dirac $\delta$-functions on $\mathcal{S}^{d-1}$. Note that a $\delta$-function $\delta_{\bm{z}_{0}}(\bm{z})$ has zero value for all $\bm{z}\in\mathcal{S}^{d-1}\setminus\{\bm{z}_{0}\}$, but $\|\delta_{\bm{z}_{0}}\|_{1}\mathrel{\mathop{:}}=\int_{\mathcal{S}^{d-1}}\delta_{\bm{z}_{0}}(\bm{z})d\mu(\bm{z})=1$. Thus, the function $g(\bm{z})$ may contain any sum of $\delta$-functions and finite-value $L^{1}$-functions. ¹¹1Alternatively, we can also interpret $g(\bm{z})$ as a signed measure (Rao & Rao, 1983) on $\mathcal{S}^{d-1}$. Then, $\delta$-functions correspond to point masses, and the condition $\|g\|_{1}<\infty$ implies that the corresponding unsigned version of the measure on $\mathcal{S}^{d-1}$ is bounded.

To see why $\mathcal{F}^{\ell_{2}}$ may correspond to the class of learnable functions, we can first examine what the learned function $\hat{f}^{\ell_{2}}$ in Eq. (4) should look like. Recall that $\mathbf{H}^{T}=[\mathbf{H}_{1}^{T}\ \cdots\ \mathbf{H}_{n}^{T}]$. Thus, $\bm{h}_{\mathbf{V}_{0},\bm{x}}\mathbf{H}^{T}=\sum_{i=1}^{n}(\bm{h}_{\mathbf{V}_{0},\bm{x}}\mathbf{H}_{i}^{T})\bm{e}_{i}^{T}$, where $\bm{e}_{i}\in\mathds{R}^{n}$ denotes the $i$-th standard basis. Combining Eq. (3) and Eq. (4), we can see that the learned function in Eq. (4) is of the form

For all $\bm{x},\bm{z}\in\mathcal{S}^{d-1}$, define $\mathcal{C}_{\bm{z},\bm{x}}^{\mathbf{V}_{0}}\mathrel{\mathop{:}}=\{j\in\{1,2,\cdots,p\}\ |\ \bm{z}^{T}\mathbf{V}_{0}[j]>0,\bm{x}^{T}\mathbf{V}_{0}[j]>0\}$, and its cardinality is given by

Then, using Eq. (1), we can show $\frac{1}{p}\bm{h}_{\mathbf{V}_{0},\bm{x}}\mathbf{H}_{i}^{T}=\bm{x}^{T}\mathbf{X}_{i}\frac{|\mathcal{C}_{\mathbf{X}_{i},\bm{x}}^{\mathbf{V}_{0}}|}{p}$. It is not hard to show that

where $\stackrel{{\scriptstyle\text{P}}}{{\rightarrow}}$ denotes converge in probability. (see Supplementary Material, Appendix D.5). Thus, if we let

then as $p\to\infty$, Eq. (3) should approach a function in $\mathcal{F}^{\ell_{2}}$. This explains why $\mathcal{F}^{\ell_{2}}$ is a candidate class of “learnable functions.” However, note that the above discussion only addresses the expressiveness of the model. It is still unclear whether any function in $\mathcal{F}^{\ell_{2}}$ can be learned with low generalization error. The following result provides the answer.

To interpret Theorem 1, we can first focus on the noiseless case, where $\bm{\epsilon}$ and Term 3 are zero. If we fix $n$ and let $p\to\infty$, then Terms 2, 5, and 6 all approach zero. We can then conclude that, in the noiseless and heavily overparameterized setting ($p\to\infty$), the generalization error will converge to a small limiting value (Term 1) that depends only on $n$. Further, this limiting value (Term 1) will converge to zero (so do Terms 4 and 7) as $n\to\infty$, i.e., when there are sufficiently many training samples. Finally, Theorem 1 holds even when there is noise.

The parameters of $q$ and $m$ can be tuned to make Eq. (10) sharper when $n$ and $p$ are large. For example, as we increase $q$, Term 1 will approach $n^{-0.5}$. Although a larger $q$ makes Terms 4, 5, and 6 bigger, as long as $n$ and $p$ are sufficiently large, those terms will still be close to 0. Similarly, if we increase $m$, then $J_{m}(n,d)$ will approach the order of $n^{2(d-1)}$. As a result, Term 3 approaches the order of $n^{2d-0.5}$ times $\|\bm{\epsilon}\|_{2}$ and the requirement $p\geq 6J_{m}(n,d)\ln\left(4n^{1+\frac{1}{m}}\right)$ approaches the order of $n^{2(d-1)}\ln n$.

Theorem 1 reveals several important insights on the generalization performance when the ground-truth function belongs to $\mathcal{F}^{\ell_{2}}$.

(i) Descent in the overparameterized region: When $p$ increases, both sides of Eq. (10) decreases, suggesting that the test error of the overfitted NTK model decreases with $p$. In Fig. 1(a), we choose a ground-truth function in $\mathcal{F}^{\ell_{2}}$ (we will explain why this function is in $\mathcal{F}^{\ell_{2}}$ later in Section 4). The test MSE of the aforementioned NTK model (green curve) confirms the overall trend⁵⁵5This curve oscillates at the early stage when $p$ is small. We suspect it is because, at small $p$, the convergence in Eq. (7) has not occurred yet, and thus the randomness in $\mathbf{V}_{0}[j]$ makes the simulation results more volatile. of descent in the overparameterized region. We note that while Arora et al. (2019) provides a generalization error upper-bound for large $p$ (i.e., Eq. (1)), the upper bound there does not capture the dependency in $p$ and thus does not predict this descent.

More importantly, we note a significant difference between the descent in Theorem 1 and that of min $\ell_{2}$-norm overfitting solutions for linear models with simple features (Belkin et al., 2018b, 2019; Bartlett et al., 2020; Hastie et al., 2019; Muthukumar et al., 2019; Liao et al., 2020; Jacot et al., 2020). For example, for linear models with Gaussian features, we can obtain (see, e.g., Theorem 2 of Belkin et al. (2019)):

where $\sigma^{2}$ denotes the variance of the noise. If we let $p\to\infty$ in Eq. (12), we can see that the MSE quickly approaches $\|f\|_{2}^{2}$, which is referred to as the “null risk” (Hastie et al., 2019), i.e., the MSE of a model that predicts zero. Note that the null-risk is at the level of the signal, and thus is quite large. In contrast, as $p\to\infty$, the test error of the NTK model converges to a value determined by $n$ and $\bm{\epsilon}$ (and is independent of the null risk). This difference is confirmed in Fig. 1(a), where the test MSE for the NTK model (green curve) is much lower than the null risk (the dashed line) when $p\to\infty$, while both the min $\ell_{2}$-norm (the red curve) and the min $\ell_{1}$-norm solutions (the blue curve) Ju et al. (2020) with Fourier features rise to the null risk when $p\to\infty$. Finally, note that the descent in Theorem 1 requires $p$ to increase much faster than $n$. Specifically, to keep Term 2 in Eq. (10) small, it suffices to let $p$ increase a little bit faster than $\Omega(n^{4d-1})$. This is again quite different from the descent shown in Eq. (12) and in other related work using Fourier and Gaussian features (Liao et al., 2020; Jacot et al., 2020), where $p$ only needs to grow proportionally with $n$.

(ii) Speed of the descent: Since Theorem 1 holds for finite $p$, it also characterizes the speed of descent. In particular, Term 2 is proportional to $p^{-\frac{1}{2}\left(1-\frac{1}{q}\right)}$, which approaches $1/\sqrt{p}$ when $q$ is large. Again, such a speed of descent is not captured in Arora et al. (2019). As we show in Fig. 1(a), the test error of the gradient descent solution under the original neural network (cyan curve) is usually quite close to that of the NTK model (green curve). Thus, our result provides useful guidance on how fast the generalization error descends with $p$ for such neural networks.

(iii) The effect of noise: Term 3 in Eq. (10) characterizes the impact of the noise $\bm{\epsilon}$, which does not decrease or increase with $p$. Notice that this is again very different from Eq. (12), i.e., results of min $\ell_{2}$-norm overfitting solutions for simple features, where the noise term $\frac{\sigma^{2}n}{p-n-1}\to 0$ when $p\to\infty$. We use Fig. 3(a) to validate this insight. In Fig. 3(a), we fix $n=50$ and plot curves of test MSE of NTK overfitting solution as $p$ increases. We let the noise $\epsilon_{i}$ in the $i$-th training sample be i.i.d. Gaussian with zero mean and variance $\sigma^{2}$. The green, red, and blue curves in Fig. 3(a) corresponds to the situation $\sigma^{2}=0$, $\sigma^{2}=0.04$, and $\sigma^{2}=0.16$, respectively. We can see that all three curves become flat when $p$ is very large, and this phenomenon implies that the gap across different noise levels does not decrease when $p\to\infty$, which is in contrast to Eq. (12).

In Fig. 3(b), we instead fix $p=20000$, and increase $n$). We plot the test MSE both for the noiseless setting (green curve) and for $\sigma^{2}=0.01$ (red curve). The difference between the two curves (dashed blue curve) then captures the impact of noise, which is related to Term 3 in Eq. (10). Somewhat surprisingly, we find that the dashed blue curve is insensitive to $n$, which suggests that Term 3 in Eq. (10) may have room for improvement.

In summary, we have shown that any ground-truth function in $\mathcal{F}^{\ell_{2}}$ leads to low generalization error for overfitted NTK models. It is then natural to ask what happens if the ground-truth function is not in $\mathcal{F}^{\ell_{2}}$. Let $\overline{\mathcal{F}^{\ell_{2}}}$ denote the closure⁶⁶6We consider the normed space of all functions in $L^{2}(\mathcal{S}^{d-1}\mapsto\mathds{R})$. Notice that although $g(\bm{z})$ in Definition 1 may not be in $L^{2}$, $f_{g}$ is always in $L^{2}$. Specifically, $f_{g}(\bm{x})$ is bounded for every $\bm{x}\in\mathcal{S}^{d-1}$ when $\|g\|_{1}<\infty$. of $\mathcal{F}^{\ell_{2}}$, and $D(f,\mathcal{F}^{\ell_{2}})$ denotes the $L^{2}$-distance between $f$ and $\mathcal{F}^{\ell_{2}}$ (i.e., the infimum of the $L^{2}$-distance from $f$ to every function in $\mathcal{F}^{\ell_{2}}$).

Intuitively, Proposition 2 (proven in Supplementary Material Appendix J) suggests that, if a ground-truth function is outside the closure of $\mathcal{F}^{\ell_{2}}$, then no matter how large $n$ is, the test error of a NTK model with infinitely many neurons cannot be small (regardless whether or not the training samples contain noise). We validate this in Fig. 1(b), where a ground-truth function is chosen outside $\overline{\mathcal{F}^{\ell_{2}}}$. The test MSE of NTK overfitting solutions (green curve) is above null risk (dashed black line) and thus is much higher compared with Fig. 1(a). We also plot the test MSE of the GD solution of the real neural network (cyan curve), which seems to show the same trend.

Comparing Theorem 1 and Proposition 2, we can clearly see that, all functions in $\mathcal{F}^{\ell_{2}}$ are learnable by the overfitted NTK model, and all functions not in $\overline{\mathcal{F}^{\ell_{2}}}$ are not.

Our expression for learnable functions in Definition 1 is still in an indirect form, i.e., through the unknown function $g(\cdot)$. In Arora et al. (2019), the authors show that all functions of the form $(\bm{x}^{T}\bm{a})^{l}$, $l\in\{0,1,2,4,6,\cdots\}$ are learnable by GD (assuming large $p$ and small step size), for a similar 2-layer network with ReLU activation that has no bias. In the following, we will show that our learnable functions in Definition 1 also have a similar form. Further, we can show that any functions of the form $(\bm{x}^{T}\bm{a})^{l}$, $l\in\{3,5,7,\cdots\}$ are not learnable. Our characterization uses an interesting connection to harmonics and filtering on $\mathcal{S}^{d-1}$, which may be of independent interest.

Towards this end, we first note that the integral form in Definition 1 can be viewed as a convolution on $\mathcal{S}^{d-1}$ (denoted by $\circledast$). Specifically, for any $f_{g}\in\mathcal{F}^{\ell_{2}}$, we can rewrite it as

where $\bm{e}\mathrel{\mathop{:}}=[0\ 0\ \cdots\ 0\ 1]^{T}\in\mathds{R}^{d}$, and $\mathbf{S}$ is a $d\times d$ orthogonal matrix that denotes a rotation in $\mathcal{S}^{d-1}$, chosen from the set $\mathsf{SO}(d)$ of all rotations. An important property of the convolution Eq. (13) is that it corresponds to multiplication in the frequency domain, similar to Fourier coefficients. To define such a transformation to the frequency domain, we use a set of hyper-spherical harmonics $\Xi_{\mathbf{K}}^{l}$ (Vilenkin, 1968; Dokmanic & Petrinovic, 2009) when $d\geq 3$, which forms an orthonormal basis for functions on $\mathcal{S}^{d-1}$. These harmonics are indexed by $l$ and $\mathbf{K}$, where $\mathbf{K}=(k_{1},k_{2},\cdots,k_{d-2})$ and $l=k_{0}\geq k_{1}\geq k_{2}\geq\cdots\geq k_{d-2}\geq 0$ (those $k_{i}$’s and $l$ are all non-negative integers). Any function $f\in L^{2}(\mathcal{S}^{d-1}\mapsto\mathds{R})$ (including even $\delta$-functions (Li & Wong, 2013)) can be decomposed uniquely into these harmonics, i.e., $f(\bm{x})=\sum_{l}\sum_{\mathbf{K}}c_{f}(l,\mathbf{K})\Xi_{\mathbf{K}}^{l}(\bm{x})$, where $c_{f}(\cdot,\cdot)$ are projections of $f$ onto the basis function. In Eq. (13), let $c_{g}(\cdot,\cdot)$ and $c_{h}(\cdot,\cdot)$ denote the coefficients corresponding to the decompositions of $g$ and $h$, respectively. Then, we must have (Dokmanic & Petrinovic, 2009)

where $\Lambda$ is some normalization constant. Notice that in Eq. (15), the coefficient for $h$ is $c_{h}(l,\bm{0})$ instead of $c_{h}(l,\mathbf{K})$, which is due to the intrinsic rotational symmetry of such convolution (Dokmanic & Petrinovic, 2009).

The above decomposition has an interesting “filtering” interpretation as follows. We can regard the function $h$ as a “filter” or “channel,” while the function $g$ as a transmitted “signal.” Then, the function $f_{g}$ in Eq. (13) and Eq. (15) can be regarded as the received signal after $g$ goes through the channel/filter $h$. Therefore, when coefficient $c_{h}(l,\bm{0})$ of $h$ is non-zero, then the corresponding coefficient $c_{f_{g}}(l,\mathbf{K})$ for $f_{g}$ can be any value (because we can arbitrarily choose $g$). In contrast, if a coefficient $c_{h}(l,\bm{0})$ of $h$ is zero, then the corresponding coefficient $c_{f_{g}}(l,\mathbf{K})$ for $f_{g}$ must also be zero for all $\mathbf{K}$.

We are now ready to characterize what functions are in $\mathcal{F}^{\ell_{2}}$. By the form of Eq. (16), for any non-negative integer $k$, any even power $(\bm{x}^{T}\bm{e})^{2k}$ is a linear combination of $\Xi_{\bm{0}}^{0},\Xi_{\bm{0}}^{2},\cdots,\Xi_{\bm{0}}^{2k}$, and any odd power $(\bm{x}^{T}\bm{e})^{2k+1}$ is a linear combination of $\Xi_{\bm{0}}^{1},\Xi_{\bm{0}}^{3},\cdots,\Xi_{\bm{0}}^{2k+1}$. By Proposition 3, we thus conclude that any function $f_{g}(\bm{x})=(\bm{x}^{T}\bm{e})^{l}$ where $l\in\{0,1,2,4,6,\cdots\}$ can be written in the form of Eq. (15) in the frequency domain, and thus are in $\mathcal{F}^{\ell_{2}}$. In contrast, any function $f(\bm{x})=(\bm{x}^{T}\bm{e})^{l}$ where $l\in\{3,5,7,\cdots\}$ cannot be written in the form of Eq. (15), and are thus not in $\mathcal{F}^{\ell_{2}}$. Further, the $\ell_{2}$-norm of any latter function will also be equal to its distance to $\mathcal{F}^{\infty}$. Therefore, the generalization-error lower-bound in Proposition 2 will apply (with $D(f,\mathcal{F}^{\ell_{2}})=\|f\|_{2}$). Finally, by Eq. (13), $\mathcal{F}^{\ell_{2}}$ is invariant under rotation and finite linear summation. Therefore, any finite sum of $(\bm{x}^{T}\bm{a})^{l}$, $l=0,1,2,4,6,\cdots$ must also belong to $\mathcal{F}^{\ell_{2}}$.

For the special case of $d=2$, the input $\bm{x}$ corresponds to an angle $\theta\in[-\pi,\pi]$, and the above-mentioned harmonics become Fourier series $\sin(k\theta)$ and $\cos(k\theta)$, $k=0,1,\cdots$. We can then get similar results that frequencies of $k\in\{0,1,2,4,6,\cdots\}$ are learnable (while others are not), which explains the learnable and not-learnable functions in Fig. 1. Details can be found in Supplementary Material, Appendix K.5.

The proof is in Supplementary Material Appendix F. Proposition 4 implies that, for fixed $n$ and $d$, when $p\to\infty$, with high probability the variance will not exceed a certain factor of the noise $\|\bm{\epsilon}\|_{2}$. In other words, the variance will not go to infinity when $p\to\infty$. The main step in the proof is to lower bound $\min\mathsf{eig}\left(\mathbf{H}\mathbf{H}^{T}\right)/p$, which is given by $1/(J_{m}(n,d)n)$. Note that this is the main place where we used the assumption that $\bm{x}$ is uniformly distributed. We expect that our main proof techniques can be generalized to other distributions (with a different expression of $J_{m}(n,d)$), which we leave for future work.

We now bound the bias part. We first study the class of ground-truth functions that can be learned with fixed $\mathbf{V}_{0}$. We refer to them as pseudo ground-truth, to differentiate them with the set $\mathcal{F}^{\ell_{2}}$ of learnable functions for random $\mathbf{V}_{0}$. They are defined with respect to the same $g(\cdot)$ function, so that we can later extend to the “real” ground-truth functions in $\mathcal{F}^{\ell_{2}}$ when considering the randomness of $\mathbf{V}_{0}$.

The reason that this class of functions may be the learnable functions for fixed $\mathbf{V}_{0}$ is similar to the discussions in Eq. (3) and Eq. (6). Indeed, using the same choice of $g(\bm{z})$ in Eq. (8), the learned function $\hat{f}^{\ell_{2}}$ in Eq. (3) at fixed $\mathbf{V}_{0}$ is always of the form in Definition 2.

The following proposition gives an upper bound of the generalization performance when the data model is based on the pseudo ground-truth and the NTK model uses exactly the same $\mathbf{V}_{0}$.

The proof is in Supplementary Material, Appendix H. Note that both the threshold of the probability event and the upper bound coincide with Term 1 and Term 4, respectively, in Eq. (10). Here we sketch the proof of Proposition 5. Based on the definition of the pseudo ground-truth, we can rewrite $f_{\mathbf{V}_{0}}^{g}$ as $f_{\mathbf{V}_{0}}^{g}(\bm{x})=\bm{h}_{\mathbf{V}_{0},\bm{x}}\Delta\mathbf{V}^{*}$, where $\Delta\mathbf{V}^{*}\in\mathds{R}^{dp}$ is given by, for all $j\in\{1,2,\cdots,p\}$, $\Delta\mathbf{V}^{*}[j]=\int_{\mathcal{S}^{d-1}}\bm{1}_{\{\bm{z}^{T}\mathbf{V}_{0}[j]>0\}}\bm{z}\frac{g(\bm{z})}{p}d\mu(\bm{z})$. From Eq. (3) and Eq. (4), we can see that the learned model is $\hat{f}^{\ell_{2}}(\bm{x})=\bm{h}_{\mathbf{V}_{0},\bm{x}}\mathbf{P}\Delta\mathbf{V}^{*}$ where $\mathbf{P}\mathrel{\mathop{:}}=\mathbf{H}^{T}(\mathbf{H}\mathbf{H}^{T})^{-1}\mathbf{H}$. Note that $\mathbf{P}$ is an orthogonal projection to the row-space of $\mathbf{H}$. Further, it is easy to show that $\|\bm{h}_{\mathbf{V}_{0},\bm{x}}\|_{2}\leq\sqrt{p}$. Thus, we have $|\hat{f}^{\ell_{2}}(\bm{x})-f_{\mathbf{V}_{0}}^{g}(\bm{x})|=|\bm{h}_{\mathbf{V}_{0},\bm{x}}(\mathbf{P}-\mathbf{I})\Delta\mathbf{V}^{*}|\leq\sqrt{p}\|(\mathbf{P}-\mathbf{I})\Delta\mathbf{V}^{*}\|_{2}$. The term $(\mathbf{P}-\mathbf{I})\Delta\mathbf{V}^{*}$ can be interpreted as the distance from $\Delta\mathbf{V}^{*}$ to the row-space of $\mathbf{H}$. Note that this distance is no greater than the distance between $\Delta\mathbf{V}^{*}$ and any point in the row-space of $\mathbf{H}$. Thus, in order to get an upper bound on $\|(\mathbf{P}-\mathbf{I})\Delta\mathbf{V}^{*}\|_{2}$, we only need to find a vector $\bm{a}\in\mathds{R}^{n}$ that makes $\|\Delta\mathbf{V}^{*}-\mathbf{H}^{T}\bm{a}\|_{2}$ as small as possible, especially when $n$ is large. Our proof uses the vector $\bm{a}$ such that its $i$-th element is $\bm{a}_{i}\mathrel{\mathop{:}}=\frac{g(\mathbf{X}_{i})}{np}$. See Supplementary Material, Appendix H for the rest of the details.

The final step is to allow $\mathbf{V}_{0}$ to be random. Given any random $\mathbf{V}_{0}$, any function $f_{g}\in\mathcal{F}^{\ell_{2}}$ can be viewed as the summation of a pseudo ground-truth function (with the same $g(\cdot)$) and a difference term. This difference can be viewed as a special form of “noise”, and thus we can use Proposition 4 to quantify its impact. Further, the magnitude of this “noise” should decrease with $p$ (because of Eq. (7)). Combining this argument with Proposition 5, we can then prove Theorem 1. See Supplementary Material, Appendix I for details.

In this paper, we studied the generalization performance of the min $\ell_{2}$-norm overfitting solution for a two-layer NTK model. We provide a precise characterization of the learnable ground-truth functions for such models, by providing a generalization upper bound for all functions in $\mathcal{F}^{\ell_{2}}$, and a generalization lower bound for all functions not in $\overline{\mathcal{F}^{\ell_{2}}}$. We show that, while the test error of the overfitted NTK model also exhibits descent in the overparameterized regime, the descent behavior can be quite different from the double descent of linear models with simple features.

There are several interesting directions for future work. First, based on Fig. 3(b), our estimation of the effect of noise could be further improved. Second, it would be interesting to explore whether the methodology can be extended to NTK model for other neural networks, e.g., with different activation functions and with more than two layers.

This work is partially supported by an NSF sub-award via Duke University (IIS-1932630), by NSF grants CNS-1717493, CNS-1901057, and CNS-2007231, and by Office of Naval Research under Grant N00014-17-1-241. The authors would like to thank Professor R. Srikant at the University of Illinois at Urbana-Champaign and anonymous reviewers for their valuable comments and suggestions.