A Statistical Analysis for Supervised Deep Learning with Exponential Families for Intrinsically Low-dimensional Data

One hypothesis for the extraordinary performance of deep learning is that most natural data have an intrinsically low-dimensional structure despite lying in a high-dimensional feature space (Pope et al.,, 2020). Under this so-called “manifold hypothesis,” the recent theoretical developments in the generalization aspects of deep learning theory literature have revealed that the excess risk for different deep learning models, especially regression (Schmidt-Hieber,, 2020; Suzuki,, 2018) and generative models (Huang et al.,, 2022; Chakraborty and Bartlett, 2024a, ; Chakraborty and Bartlett, 2024b, ) exhibit a decay pattern that depends only on the intrinsic dimension of the data. Notably, Nakada and Imaizumi, (2020), Huang et al., (2022) and Chakraborty and Bartlett, 2024a showed that the excess risk decays as $\mathcal{O}(n^{-1/\mathcal{O}(d_{\mu})})$, where $d_{\mu}$ denotes the Minkowski dimension of the underlying distribution. For a supervised learning setting, this phenomenon has been proved for various deep regression models with additive Gaussian noise (Schmidt-Hieber,, 2020; Nakada and Imaizumi,, 2020; Suzuki,, 2018; Suzuki and Nitanda,, 2021).

Most of the aforementioned studies aim to describe this inherent dimensionality by utilizing the concept of the (upper) Minkowski dimension of the data’s underlying support. However, it is worth noting that the Minkowski dimension primarily focuses on quantifying the rate of growth in the covering number of the support while neglecting to account for situations where the distribution may exhibit a higher concentration of mass within specific sub-regions of this support. Thus, the Minkowski dimension often overestimates the intrinsic dimension of the data distribution, resulting in slower rates of statistical convergence. On the other hand, some works (Chen et al.,, 2022, 2019; Jiao et al.,, 2021) try to impose a smooth Riemannian manifold structure for this support and characterize the rate through the dimension of this manifold. However, this assumption is not only very strong and difficult to verify, but also ignores the fact that the data can be concentrated only on some sub-regions and can be thinly spread over the remainder, again resulting in an over-estimate. In contrast, recent insights from the optimal transport literature have introduced the concept of the Wasserstein dimension (Weed and Bach,, 2019), which overcomes these limitations and offers a more accurate characterization of convergence rates when estimating a distribution through the empirical measure. Furthermore, recent advancements in this field have led to the introduction of the entropic dimension (Chakraborty and Bartlett, 2024b, ), which builds upon seminal work by Dudley, (1969) and can be employed to describe the convergence rates for Bidirectional Generative Adversarial Networks (BiGANs) (Donahue et al.,, 2017). Remarkably, the entropic dimension is no larger than the Wasserstein and Minkowski dimensions, resulting in faster convergence rates. However, it is not known whether this faster rate of convergence extends beyond Generative Adversarial Networks (GANs) and their variants to supervised learning problems.

In this paper, we provide a statistical framework aimed at understanding deep supervised learning. Our approach involves modeling the conditional distribution of the response variable given the explanatory variable as a member of an exponential family with a smooth mean function. This framework accommodates a wide spectrum of scenarios, including standard regression and classification tasks, while also providing a statistical foundation for handling complex dependencies in real data settings. In this framework, the maximum likelihood estimates can be viewed as minimizing the canonical Bregman divergence loss between the predicted values and the actual responses. When the explanatory variable has a bounded density with respect to the $d$-dimensional Lebesgue measure, our analysis reveals that deep networks employing ReLU activation functions can achieve a test error on the order of $\tilde{\mathcal{O}}(n^{-2\beta/(2\beta+d)})$ provided that appropriately sized networks are selected. Here $\beta$ denotes the Hölder smoothness of the true mean response function. This generalizes the known results in the literature for additive regression with Gaussian noise.

Another aspect overlooked by the current literature is that even when the explanatory variable is absolutely continuous, the rate of convergence of the sample estimator often exponentially increases with the ambient feature dimension, making the upper bound on the estimation error weak for high-dimensional data. In this paper, we prove that if the explanatory variable has a bounded density, the dependence, in terms of the ambient feature dimension, is not exponential but at most polynomial. Furthermore, we show that the derived rates for the test error are roughly minimax optimal, meaning that one cannot achieve a better rate of convergence through any estimator except for only potentially improving a logarithmic dependence on $n$. Lastly, when the data has an intrinsically low dimensional structure, we show that the test error can be improved to achieve a rate of roughly $\tilde{\mathcal{O}}(n^{-2\beta/(2\beta+\bar{d}_{2\beta}(\lambda))})$, where $\bar{d}_{2\beta}(\lambda)$ denotes the $2\beta$-entropic dimension (see Definition 11) of $\lambda$, the distribution of the explanatory variables, thus, improving upon the rates in the current literature. This result not only extends the framework beyond additive Gaussian noise regression models but also improves upon the existing rates (Nakada and Imaizumi,, 2020; Schmidt-Hieber,, 2020; Chen et al.,, 2022). The main results of this paper are summarized as follows:

• •

  In Theorem 8, we demonstrate that when the explanatory variable has a bounded density, the test error for the learning problem scales as $\tilde{\mathcal{O}}\left(d^{\frac{2\lfloor\beta\rfloor(\beta+d)}{2\beta+d}}n^{-\frac{2\beta}{2\beta+d}}\right)$, showing explicit dependence on the problem dimension ($d$) and the number of samples ($n$)

• •

  Theorem 10 establishes that the minimax rates scale as $\tilde{\mathcal{O}}\left(n^{-\frac{2\beta}{2\beta+d}}\right)$, ensuring that deep learners can attain the minimax optimal rate when network sizes are appropriately chosen. Notably, this theorem recovers the seminal results of Yang and Barron, (1999) as a special case.

• •

  When the explanatory variable has an intrinsically low dimensional structure, we show that deep supervised learners can effectively achieve an error rate of $\tilde{\mathcal{O}}\left(n^{-\frac{2\beta}{2\beta+\bar{d}_{2\beta}(\lambda)}}\right)$ in Theorem 12. This result provides the fastest known rates for deep supervised learners and encompasses many recent findings as special cases (Nakada and Imaizumi,, 2020; Chen et al.,, 2022) for additive regression models.

• •

  In the process, in Lemma 21 we are able to improve upon the recent $\mathbb{L}_{p}$-approximation results on ReLU networks, which might be of independent interest.

The remainder of the paper is organized as follows. After discussing the necessary background in Section 2, we discuss the exponential family learning framework in Section 3. Under this framework, we derive the learning rates (Theorem 8) when the explanatory variable is absolutely continuous in Section 4 and show that it is minimax optimal (Theorem 10). Next, we analyze the error rate (Theorem 12) when the explanatory variable has an intrinsically low dimensional structure in Section 5. The proofs of the main results are sketched in Section 6, followed by concluding remarks in Section 7.

This section recalls some of the notation and background that we need. We say $A_{n,d}\precsim B_{n,d}$ (for $A,\,B\geq 0$) if there exists an absolute constant $C>0$ (independent of $n$ and $d$), such that $A_{n,d}\leq CB_{n,d}$, for all $n,d$. Similarly, for non-negative functions $f$ and $g$, we say $f(x)\precsim_{x}g(x)$ if there exists a constant $C$, which is independent of $x$ such that $f(x)\leq Cg(x)$, for all $x$. For any function $f:\mathcal{S}\to\mathop{\mathbb{R}}\nolimits$, and any measure $\gamma$ on $\mathcal{S}$, let $\|f\|_{\mathbb{L}_{p}(\gamma)}:=\left(\int_{\mathcal{S}}|f(x)|^{p}d\gamma(x)\right)^{1/p}$, if $0<p<\infty$. Also let, $\|f\|_{\mathbb{L}_{\infty}(\gamma)}:=\operatorname{ess\,sup}_{x\in\text{supp}(\gamma)}|f(x)|$. We say $A_{n}=\tilde{\mathcal{O}}(B_{n})$ if $A_{n}\leq B_{n}\times\log^{C}(en)$, for some factor constant $C>0$. Moreover, $x\vee y=\max\{x,y\}$ and $x\wedge y=\min\{x,y\}$.

It is evident that $d_{\phi}(x,y)\geq 0$ holds for all $x,y\in\mathop{\mathbb{R}}\nolimits^{p}$ due to the fact that $\phi(x)\geq\phi(y)+\langle\nabla\phi(y),x-y\rangle$ is equivalent to the convex nature of the function $\phi$. From a geometric standpoint, one can conceptualize $d_{\phi}(x\|y)$ as the separation between $\phi(x)$ and the linear approximation of $\phi(x)$ centered around $\phi(y)$. Put simply, this can be described as the distance between $\phi(x)$ and the value obtained by evaluating the tangent line to $\phi(y)$ at the point $x$. Some prominent examples of Bregman divergences include the squared Euclidean distance, Kullback-Leibler (KL) divergence, Mahalanobis distance, etc. We refer the reader to Banerjee et al., (2005, Table 1) for more examples of the Bregman family. Bregman divergences have a direct association with standard exponential families, as elaborated in the upcoming section, rendering them particularly suitable for modeling and learning from various common data types that originate from exponential family distributions.

To discuss our framework, let us first recall the definition of exponential families (Lehmann and Casella,, 2006, Chapter 1, Section 5). We suppose that $\boldsymbol{\theta}\in\Theta$ is the natural parameter. We say that $\boldsymbol{X}$ is distributed according to an exponential family, $\mathscr{F}_{\Psi}$ if the density of $\boldsymbol{X}$ w.r.t. some dominating measure $\nu$, is given by,

$$p_{\Psi,\theta}(d\boldsymbol{x})=h(\boldsymbol{x})\exp\left(\langle\boldsymbol{\theta},T(\boldsymbol{x})\rangle-\Psi(\boldsymbol{\theta})\right)\nu(d\boldsymbol{x}).$$

Here, $T(\cdot)$ is called the natural statistic. Often, it is assumed that the exponential family is expressed in its regular form, which means that the components of $T(\cdot)$ are affinely independent, i.e. there exists no $\boldsymbol{\upsilon}$, such that $\langle\boldsymbol{\upsilon},T(\boldsymbol{x})\rangle=c$ (a constant), for all $\boldsymbol{x}$. Popular examples of exponential families include Gaussian, binomial, Poisson, and exponential distributions. Given an exponential family, one can express it in its natural form. Formally,

It is well known that $\boldsymbol{\mu}(\boldsymbol{\theta}):=\mathbb{E}_{\boldsymbol{X}\sim p_{\Psi,\boldsymbol{\theta}}}\boldsymbol{X}=\nabla\Psi(\boldsymbol{\theta})$. For simplicity, we assume that $\Psi$ is proper, closed, convex, and differentiable. The conjugate of $\Psi$, denoted as $\phi$ is defined as, $\phi(\boldsymbol{\mu})=\sup_{\boldsymbol{\theta}\in\Theta}\left\{\langle\boldsymbol{\theta},\boldsymbol{\mu}\rangle-\Psi(\boldsymbol{\theta})\right\}$. It is well known (Banerjee et al.,, 2005, Theorem 4) that $p_{\Psi,\boldsymbol{\theta}}$ can be expressed as,

$$p_{\Psi,\boldsymbol{\theta}}(d\boldsymbol{x})=\exp(-d_{\phi}(\boldsymbol{x}\|\boldsymbol{\mu}(\boldsymbol{\theta}))b_{\phi}(\boldsymbol{x})\nu(d\boldsymbol{x}),$$

(2)

where $d_{\phi}(\cdot\|\cdot)$ denotes the Bergman divergence corresponding to $\phi$. Here, $b_{\phi}(\boldsymbol{x})=\exp(\phi(\boldsymbol{x}))h(\boldsymbol{x})$. In this paper, we are interested in the supervised learning problem when the response, given the explanatory variable, is distributed according to an exponential family. For simplicity, we assume that the responses are real-valued. We assume that there exists a “true” predictor function, $f_{0}:\mathop{\mathbb{R}}\nolimits^{d}\to\mathop{\mathbb{R}}\nolimits$, such that

$$y|\boldsymbol{x}\sim p_{\Psi,f_{0}(\boldsymbol{x})}\quad\text{and}\quad\boldsymbol{x}\sim\lambda.$$

Thus, the joint distribution of $(\boldsymbol{X},Y)$ is given by,

$$\mathscr{P}(d\boldsymbol{x},dy)\propto\exp(-d_{\phi}(y\|\mu(f_{0}(\boldsymbol{x})))b_{\phi}(y)\lambda(d\boldsymbol{x})\nu(dy).$$

(3)

By definition, we observe that $\mathbb{E}(y|\boldsymbol{x})=\mu(f_{0}(\boldsymbol{x}))$. We will assume that the data is independent and identically distributed according to the distribution $\mathscr{P}$. We also assume that the distribution of $\boldsymbol{x}$ is bounded in the compact set, $[0,1]^{d}$. Formally,

In the classical statistics literature, one estimates $f_{0}$ by finding its maximum likelihood estimates (m.l.e.) as,

$$\mathop{\rm argmax}_{f\in\mathcal{F}}\frac{1}{n}\sum_{i=1}^{n}\log\mathscr{P}(\boldsymbol{x}_{i},y_{i}).$$

Plugging in the form of $\mathscr{P}$ as in (3), it is easy to see that the above optimization problem is equivalent to

$$\mathop{\rm argmin}_{f\in\mathcal{F}}\frac{1}{n}\sum_{i=1}^{n}d_{\phi}\left(y_{i}\|\mu(f(\boldsymbol{x}_{i}))\right)$$

(4)

Here, $\mu:\mathop{\mathbb{R}}\nolimits\to\mathop{\mathbb{R}}\nolimits$ is known as the link function. In practice, we take $\mathcal{F}$ to be some sort of neural network class, with the final output passing through the activation function $\mu$. The empirical minimizer of (4) is denoted as $\hat{f}$. To show that this framework covers a wide range of supervised learning problems, we consider the classical example for the case when $y|\boldsymbol{x}\sim\text{Normal}(f_{0}(\boldsymbol{x}),\sigma^{2})$, for some unknown $\sigma^{2}$. In this case, it is well known that $\mu(\cdot)$ is the identity map and $d_{\phi}(\cdot\|\cdot)$ becomes the squared Euclidean distance. Thus, the m.l.e. problem (4) becomes the classical regression problem, i.e. $\mathop{\rm argmin}_{f\in\mathcal{F}}\frac{1}{n}\sum_{i=1}^{n}(y_{i}-f(\boldsymbol{x}_{i}))^{2}$.

Another example is the case of logistic regression. We assume that $y|\boldsymbol{x}$ is a Bernoulli random variable. This makes $\mu(z)=\frac{1}{1+e^{-z}}$, i.e. the sigmoid activation function. Furthermore, an easy calculation (Banerjee et al.,, 2005, Table 2) shows that $d_{\phi}(x\|y)=x\log\left(\frac{x}{y}\right)+(1-x)\log\left(\frac{1-x}{1-y}\right)$. Plugging in the values of $\mu$ and $d_{\phi}$ into (4), we note that the estimation problem becomes,

$$\mathop{\rm argmin}_{f\in\mathcal{F}}-\frac{1}{n}\sum_{i=1}^{n}\left(y_{i}\log\circ\,\text{sigmoid}(f(\boldsymbol{x}_{i}))+(1-y_{i})\log\circ(1-\text{sigmoid}(f(\boldsymbol{x}_{i})))\right).$$

Here, $\operatorname{sigmoid}(t)=1/(1+e^{-t})$ denotes the sigmoid activation function. Thus, the problem reduces to the classical two-class learning problem with the binary cross-entropy loss and with sigmoid activation at the final layer. For simplicity, we assume that all activations, excluding that of the final layer of $f$, are realized by the ReLU activation function. The choice of ReLU activation is a natural choice for practitioners and enables us to harness the approximation theory of ReLU networks developed throughout the recent literature (Yarotsky,, 2017; Uppal et al.,, 2019). However, using a leaky ReLU network will also result in a similar analysis, changing only the constants in the main theorems.

To facilitate the theoretical analysis, we will assume that the problem is smooth in terms of the learning function $f_{0}$. As a notion of smoothness, we will use Hölder smoothness. This has been a popular choice in the recent literature (Nakada and Imaizumi,, 2020; Schmidt-Hieber,, 2020; Chen et al.,, 2022) and covers a vast array of functions commonly modeled in the literature.

We make the additional assumption that the function $\Psi$ is well-behaved. In particular, we assume that $\Psi$ possesses both smoothness and strong convexity properties. It is important to note that these assumptions are widely employed in the existing literature (Telgarsky and Dasgupta,, 2013; Paul et al.,, 2021). Though A3 is not applicable for the classification problem, as in that case, $\Psi(x)=x\ln x+(1-x)\ln(1-x)$, which does not satisfy A3. However for all practical purposes, one clips the output network (which is often done in practice to ensure smooth training), i.e. ensures that $\epsilon\leq f,f_{0}\leq 1-\epsilon$, for some positive $\epsilon$, A3 is satisfied. The assumption is formally stated as follows.

A direct implication of A3 is that by Kakade et al., (2009, Theorem 6), $\phi$ is $\tau_{2}$-smooth and $\tau_{1}$- strongly convex. Here $\tau_{i}=1/\sigma_{i}$. Also, since $\Psi$ is $\sigma_{1}$-smooth, $\mu(\cdot)=\nabla\Psi(\cdot)$ is $\sigma_{1}$-Lipschitz. This fact will be useful for the proofs of the main results. In the subsequent sections, under the above assumptions, we derive probabilistic error bounds for the excess risk of $\hat{f}$.

We begin the analysis of the test error for the problem (4) when $\lambda$, the distribution of the explanatory variable has a bounded density on $[0,1]^{d}$. First note that the excess risk is upper bounded by the estimation error for $f_{0}$ in the $\mathbb{L}_{2}(\lambda)$-norm. The excess risk for the problem is given by

$$\mathfrak{R}(\hat{f})=\mathbb{E}_{(y,\boldsymbol{x})\sim\mathscr{P}}d_{\phi}(y\|\mu(\hat{f}(\boldsymbol{x})))-\mathbb{E}_{(y,\boldsymbol{x})\sim\mathscr{P}}d_{\phi}(y\|\mu(f_{0}(\boldsymbol{x}))).$$

The following lemma ensures that $\mathfrak{R}(\hat{f})\asymp\|\hat{f}-f_{0}\|_{\mathbb{L}_{2}(\lambda)}^{2}$ and hence, it is enough to prove bounds on $\|\hat{f}-f_{0}\|_{\mathbb{L}_{2}(\lambda)}^{2}$ to derive upper and lower bounds on the excess risk.

As already mentioned, we assume that $\lambda$ admits a density w.r.t. the Lebesgue measure, and this density is upper bounded. Formally,

Under Assumptions A1–4, we observe that with high probability, $\|\hat{f}-f_{0}\|^{2}_{\mathbb{L}_{2}(\lambda)}$ and thereby, $\mathfrak{R}(\hat{f})$ scales at most as $\tilde{\mathcal{O}}\left(d^{2\lfloor\beta\rfloor}n^{-\frac{2\beta}{2\beta+d}}\right)$, ignoring $\log$-terms in $n$, for large $n$, if the network sizes are chosen properly. This rate of decrease aligns consistently with prior findings reported by Nakada and Imaizumi, (2020) for additive Gaussian-noise regression. It is important to underscore that the existing literature predominantly investigates the rate of decrease in $\mathfrak{R}(\hat{f})$ solely with regard to the sample size $n$, overlooking terms dependent upon the data dimensionality. These dimension-dependent terms harbor the potential for exponential growth with respect to the dimensionality of the explanatory variables, and may therefore attain substantial magnitudes, making such bounds inefficient in high-dimensional statistical learning contexts. This analysis shows that the dependence in $d$ is not exponential and can be made to increase at a polynomial rate only under the assumption of the existence of a bounded density of the explanatory variable. The main upper bound for this case is stated in Theorem 8, with a proof outline appearing in Section 6.1.

From the bound on the network size, i.e. $W$ in Theorem 8, it is clear that when $f_{0}$ is smooth i.e. for large $\beta$, one requires a network of smaller size compared to when $f_{0}$ is less smooth i.e. when $\beta$ is small. Similarly, in cases where the dimensionality of the explanatory variables, represented by $d$ is substantial, a larger network is required as compared to situations where $d$ is relatively small. This observation aligns with the intuitive expectation that solving more intricate and complex problems in higher dimensions demands the utilization of larger networks.

The high probability bound in Theorem 8 ensures that the expected test error also scales with the same rate of convergence. This result is shown in Corollary 9

To understand whether deep supervised learning can achieve the optimal rates for the learning problem, we derive the minimax rates for estimating $f_{0}$. The minimax expected risk for this problem, when one has access to $n$ i.i.d. samples $\{(\boldsymbol{x}_{i},y_{i})\}_{i\in[n]}$ from (3) ($f_{0}$ replaced with $f$) is given by

$$\mathfrak{M}_{n}=\inf_{\hat{f}}\sup_{f\in\mathscr{H}^{\beta}(\mathop{\mathbb{R}}\nolimits^{d},\mathop{\mathbb{R}}\nolimits,C)}\mathbb{E}_{f}\|\hat{f}-f\|_{\mathbb{L}_{2}(\lambda)}^{2},$$

With the notation $\mathbb{E}_{f}(\cdot)$ we denote the expectation w.r.t. the measure (3), with $f_{0}$ replaced with $f$. Here the infimum is taken over all measurable estimates of $\hat{f}$, based on the data. Minimax lower bounds are used to understand the theoretical limits of any statistical estimation problem. The aim of this analysis is to show that deep learning with ReLU networks for the exponential family dependence is (almost) minimax optimal. To facilitate the theoretical analysis, we assume that the density of $\lambda$ is lower bounded by a positive constant. Formally,

Theorem 10 provides a characterization of this minimax lower bound for estimating $f$. It is important to note that the seminal works of Yang and Barron, (1999) for the normal-noise regression problem is a special case of Theorem 10.

Thus, from Theorems 8 and 10 it is clear that deep supervised estimators for the exponential family dependence can achieve this minimax optimal rate with high probability, barring an excess poly-log factor of $n$.

Frequently, it is posited that real-world data, such as vision data, resides within a lower-dimensional structure embedded in a high-dimensional feature space (Pope et al.,, 2020). To quantify this intrinsic dimensionality of the data, researchers have introduced various measures of the effective dimension of the underlying probability distribution assumed to generate the data. Among these approaches, the most commonly used ones involve assessing the rate of growth of the covering number, in a logarithmic scale, for most of the support of this data distribution. Let us consider a compact Polish space denoted as $(\mathcal{S},\varrho)$, with $\gamma$ representing a probability measure defined on it. For the remainder of this paper, we will assume that $\varrho$ corresponds to the $\ell_{\infty}$-norm. The simplest measure of the dimension of a probability distribution is the upper Minkowski dimension of its support, defined as follows:

$$\overline{\text{dim}}_{M}(\gamma)=\limsup_{\epsilon\downarrow 0}\frac{\log\mathcal{N}(\epsilon;\text{supp}(\gamma),\ell_{\infty})}{\log(1/\epsilon)}.$$

This dimensionality concept relies solely on the covering number of the support and does not assume the existence of a smooth mapping to a lower-dimensional Euclidean space. Consequently, it encompasses not only smooth Riemannian manifolds but also encompasses highly non-smooth sets like fractals. The statistical convergence properties of various estimators concerning the upper Minkowski dimension have been extensively explored in the literature. Kolmogorov and Tikhomirov, (1961) conducted a comprehensive study on how the covering number of different function classes depends on the upper Minkowski dimension of the support. Recently, Nakada and Imaizumi, (2020) demonstrated how deep learning models can leverage this inherent low-dimensionality in data, which is also reflected in their convergence rates. Nevertheless, a notable limitation associated with utilizing the upper Minkowski dimension is that when a probability measure covers the entire sample space but is concentrated predominantly in specific regions, it may yield a high dimensionality estimate, which might not accurately reflect the underlying dimension.

To overcome the aforementioned difficulty, as a notion of the intrinsic dimension of a measure $\gamma$, Chakraborty and Bartlett, 2024b introduced the notion of $\alpha$-entropic dimension of a measure. Before we proceed, we recall the $(\epsilon,\tau)$-cover of a measure (Posner et al.,, 1967) as: $\mathscr{N}_{\epsilon}(\gamma,\tau)=\inf\{\mathcal{N}(\epsilon;S,\varrho):\gamma(S)\geq 1-\tau\},$ i.e. $\mathscr{N}_{\epsilon}(\gamma,\tau)$ counts the minimum number of $\epsilon$-balls required to cover a set $S$ of probability at least $1-\tau$.

This notion extends Dudley’s notion of entropic dimension (Dudley,, 1969) to characterize the convergence rate for the BiGAN problem (Donahue et al.,, 2017). Chakraborty and Bartlett, 2024b showed that the entropic dimension is not larger than the upper Minkowski dimension and the upper Wasserstein dimension (Weed and Bach,, 2019). Furthermore, strict inequality holds even for simplistic examples for measures on the unit hypercube. We refer the reader to Section 3 of Chakraborty and Bartlett, 2024b . Chakraborty and Bartlett, 2024b showed that the entropic dimension is a more efficient way of characterizing the intrinsic dimension of the data distributions compared to popular measures such as the upper Minkowski dimension or the Wasserstein dimension (Weed and Bach,, 2019) as the entropic dimension enables us to derive a faster rate of convergence of the estimates. Importantly, $\bar{d}_{\alpha}(\gamma)\leq\overline{dim}_{M}(\gamma)$, with strict inequality holding, even for simplistic cases (see examples 10 and 11 of Chakraborty and Bartlett, 2024b, ).

As an intrinsically low-dimensional probability measure is not guaranteed to be dominated by the Lebesgue measure, we remove Assumption A4. Under only Assumptions A1–3, if the network sizes are properly chosen, the rate of convergence of $\hat{f}$ to $f_{0}$ under the $\mathbb{L}_{2}(\lambda)$-norm decays at a rough rate of $\tilde{\mathcal{O}}\left(n^{-2\beta/(2\beta+\bar{d}_{2\beta}(\lambda))}\right)$, as shown by Theorem 12.

Since the normal-noise regression model with $\beta$-Hölder $f_{0}$ is a special case of our model in (3), Theorem 12 derives a faster rate compared to Nakada and Imaizumi, (2020, Theorem 7), who show a rate of $\tilde{\mathcal{O}}\left(n^{-\frac{2\beta}{2\beta+\overline{\text{dim}}_{M}(\lambda)}}\right)$. This is because the upper Minkowski dimension is at least the $2\beta$-entropic dimension by Chakraborty and Bartlett, 2024b (, Proposition 8(c)), i.e. $\bar{d}_{2\beta}(\lambda)\leq\overline{\text{dim}}_{M}(\lambda)$.

An immediate corollary of Theorem 12 is that the expected test-error rate follows the same rate of decay. The proof of this result can be done following the proof of Corollary 9.

One can state that a rate similar to that observed in Theorem 12 holds when the support of $\lambda$ is regular. We recall the definition (Weed and Bach,, 2019, Definition 6) of regular sets in $[0,1]^{d}$ as follows.

Examples of regular sets include compact $\tilde{d}$-dimensional differentiable manifolds; nonempty, compact convex sets spanned by an affine space of dimension $\tilde{d}$; the relative boundary of a nonempty, compact convex set of dimension $\tilde{d}+1$; or a self-similar set with similarity dimension $\tilde{d}$. When the support of $\lambda$ is $\tilde{d}$-regular, it can be shown that $\bar{d}_{\alpha}(\lambda)\leq\tilde{d}$. Formally,

Thus, applying Theorem 12 and Lemma 15, we note that $\|\hat{f}-f_{0}\|_{\mathbb{L}_{2}(\lambda)}^{2}$ decays at most at a rate of $\tilde{\mathcal{O}}\left(n^{-2\beta/(2\beta+\tilde{d})}\right)$, resulting in the following corollary.

Since compact $\tilde{d}$-dimensional differentiable manifolds are a special case of $\tilde{d}$-regular sets, Corollary 16 recovers the results by Chen et al., (2022) as a special case i.e., an additive Gaussian-noise regression model. Importantly, this recovery is achieved without imposing assumptions about uniform sharpness on the manifold, as done by Chen et al., (2022, Assumption 2).

This section discusses the proof of the main results of this paper, i.e, Theorems 8, 10 and 12, with proofs of auxiliary supporting lemmas appearing in the appendix. The proof of the main upper bounds (Theorems 8 and 12) are presented in Section 6.1, while the minimax lower bound is proved in Section 6.2.