Differentiable Neural Networks with RePU Activation: with Applications to Score Estimation and Isotonic Regression

Score estimation is an important approach to distribution learning and score function plays a central role in the diffusion-based generative learning (Song et al., 2021; Block et al., 2020; Ho et al., 2020; Lee et al., 2022). Let $X\sim p_{0}$, where $p_{0}$ is a probability density function supported on $\mathbb{R}^{d}$, then $d$-dimensional score function of $p_{0}$ is defined as $s_{0}(x)=\nabla_{x}\log p_{0}(x)$, where $\nabla_{x}$ is the vector differential operator with respect to the input $x$.

A score matching estimator (Hyvärinen and Dayan, 2005) is obtained by solving the minimization problem

$$\min_{s\in\mathcal{F}}\frac{1}{2}\mathbb{E}_{X}\|s(X)-s_{0}(X)\|^{2}_{2},$$

(1)

where $\|\cdot\|_{2}$ denotes the Euclidean norm, $\mathcal{F}$ is a prespecified class of functions, often referred to as a hypothesis space. However, this objective function is computationally infeasible because $s_{0}$ is unknown. Under some mild conditions given in Assumption 9 in Section 4, it can be shown that (Hyvärinen and Dayan, 2005)

$$\frac{1}{2}\mathbb{E}_{X}\|s(X)-s_{0}(X)\|^{2}_{2}=J(s)+\frac{1}{2}\mathbb{E}_{X}\|s_{0}(X)\|_{2}^{2},$$

(2)

with

$$J(s):=\mathbb{E}_{X}\left[{\rm tr}(\nabla_{x}s(X))+\frac{1}{2}\|s(X)\|_{2}^{2}\right],$$

(3)

where $\nabla_{x}s(x)$ denotes the Jacobian matrix of $s(x)$ and ${\rm tr}(\cdot)$ the trace operator. Since the second term on the right side of (2), $\mathbb{E}_{X}\|s_{0}(X)\|_{2}^{2}/2$, does not involve $s$, it can be considered a constant. Therefore, we can just use $J$ in (3) as an objective function for estimating the score function $s_{0}$. When a random sample is available, we use a sample version of $J$ as the empirical objective function. Since $J$ involves the partial derivatives of $s(x)$, we need to compute the derivatives of the functions in $\mathcal{F}$ during estimation. And we need to analyze the properties of $\mathcal{F}$ and their derivatives to develop the learning theories. In particular, if we take $\mathcal{F}$ to be a class of deep neural network functions, we need to study the properties of their derivatives in terms of estimation and approximation.

Isotonic regression is a technique that fits a regression model to observations such that the fitted regression function is non-decreasing (or non-increasing). It is a basic form of shape-constrained estimation and has applications in many areas, such as epidemiology (Morton-Jones et al., 2000), medicine (Diggle et al., 1999; Jiang et al., 2011), econometrics (Horowitz and Lee, 2017), and biostatistics (Rueda et al., 2009; Luss et al., 2012; Qin et al., 2014).

Consider a regression model

$$Y=f_{0}(X)+\epsilon,$$

(4)

where $X\in\mathcal{X}\subseteq\mathbb{R}^{d},$ $Y\in\mathbb{R},$ and $\epsilon$ is an independent noise variable with $\mathbb{E}(\epsilon)=0$ and ${\rm Var}(\epsilon)\leq\sigma^{2}.$ In (4), $f_{0}$ is the underlying regression function, which is usually assumed to belong to certain smooth function class.

In isotonic regression, $f_{0}$ is assumed to satisfy a monotonicity property as follows. Let $\preceq$ denote the binary relation “less than” in the partially ordered space $\mathbb{R}^{d}$, i.e., $u\preceq v$ if $u_{j}\leq v_{j}$ for all $j=1,\ldots,d,$ where $u=(u_{1},\ldots,u_{d})^{\top},v=(v_{1},\ldots,v_{d})^{\top}\in\mathcal{X}\subseteq\mathbb{R}^{d}$. In isotonic regression, the target regression function $f_{0}$ is assumed to be coordinate-wisely non-decreasing on $\mathcal{X}$, i.e., $f_{0}(u)\leq f_{0}(v)$ if $u\preceq v.$ The class of isotonic regression functions on $\mathcal{X}\subseteq\mathbb{R}^{d}$ is the set of coordinate-wisely non-decreasing functions

$$\mathcal{F}_{0}:=\{f:\mathcal{X}\to\mathbb{R},f(u)\leq f(v){\rm\ if\ }u\preceq v,\text{ where }\mathcal{X}\subset\mathbb{R}^{d}\}.$$

The goal is to estimate the target regression function $f_{0}$ under the constraint that $f_{0}\in\mathcal{F}_{0}$ based on an observed sample $S:=\{(X_{i},Y_{i})\}_{i=1}^{n}$. For a possibly random function $f:\mathcal{X}\to\mathbb{R}$, let the population risk be

$$\mathcal{R}(f)=\mathbb{E}|Y-f(X)|^{2},$$

(5)

where $(X,Y)$ follows the same distribution as $(X_{i},Y_{i})$ and is independent of $f$. Then the target function $f_{0}$ is the minimizer of the risk $\mathcal{R}(f)$ over $\mathcal{F}_{0},$ i.e.,

$$f_{0}\in\arg\min_{f\in\mathcal{F}_{0}}\mathcal{R}(f).$$

(6)

The empirical version of (6) is a constrained minimization problem, which is generally difficult to solve directly. In light of this, we propose a penalized approach for estimating $f_{0}$ based on the fact that, for a smooth $f_{0}$, it is increasing with respect to the $j$th argument $x_{j}$ if and only if its partial derivative with respect to $x_{j}$ is nonnegative. Let $\dot{f}_{j}(x)=\partial f(x)/\partial x_{j}$ denote the partial derivative of $f$ with respective to $x_{j},j=1,\ldots,d.$ We propose the following penalized objective function

$$\mathcal{R}^{\lambda}(f)=\mathbb{E}|Y-f(X)|^{2}+\frac{1}{d}\sum_{j=1}^{d}\lambda_{j}\mathbb{E}\{\rho(\dot{f}_{j}(X))\},$$

(7)

where $\lambda=\{\lambda_{j}\}_{j=1}^{d}$ with $\lambda_{j}\geq 0,j=1,\ldots,d$, are tuning parameters, $\rho(\cdot):\mathbb{R}\to[0,\infty)$ is a penalty function satisfying $\rho(x)\geq 0$ for all $x$ and $\rho(x)=0$ if $x\geq 0$. Feasible choices include $\rho(x)=\max\{-x,0\}$, $\rho(x)=[\max\{-x,0\}]^{2}$ and more generally $\rho(x)=h(\max\{-x,0\})$ for a Lipschitz function $h$ with $h(0)=0$.

The objective function (7) turns the constrained isotonic regression problem into a penalized regression problem with penalties on the partial derivatives of the regression function. Therefore, if we analyze the learning theory of estimators in (7) using neural network functions, we need to study the partial derivatives of the neural network functions in terms of their generalization and approximation properties.

It is worth mentioning that an advantage of our penalized formulation over the hard-constrained isotonic regressions is that the resulting estimator remains consistent with proper tuning when the underlying regression function is not monotonic. Therefore, our proposed method has a robustness property against model misspecification. We will discuss this point in detail in Section 5.2.

A commonality between the aforementioned two quite different problems is that they both involve the derivatives of the target function, in addition to the target function itself. When deep neural networks are used to parameterize the hypothesis space, the derivatives of deep neural networks must be considered. To study the statistical learning theory for these deep neural methods, it requires the knowledge of the complexity and approximation properties of deep neural networks along with their derivatives.

Complexities of deep neural networks with ReLU and piece-wise polynomial activation functions have been studied by Anthony and Bartlett (1999) and Bartlett et al. (2019). Generalization bounds in terms of the operator norm of neural networks have also been obtained by several authors (Neyshabur et al., 2015; Bartlett et al., 2017; Nagarajan and Kolter, 2019; Wei and Ma, 2019). These generalization results are based on various complexity measures such as Rademacher complexity, VC-dimension, Pseudo-dimension, and norm of parameters. These studies shed light on the complexity and generalization properties of neural networks themselves, however, the complexities of their derivatives remain unclear.

The approximation power of deep neural networks with smooth activation functions has been considered in the literature. The universality of sigmoidal deep neural networks has been established by Mhaskar (1993) and Chui et al. (1994). In addition, the approximation properties of shallow RePU-activated networks were analyzed by Klusowski and Barron (2018) and Siegel and Xu (2022). The approximation rates of deep RePU neural networks for several types of target functions have also been investigated. For instance, Li et al. (2019, 2020), and Ali and Nouy (2021) studied the approximation rates for functions in Sobolev and Besov spaces in terms of the $L_{p}$ norm, Duan et al. (2021), Abdeljawad and Grohs (2022) studied the approximation rates for functions in Sobolev space in terms of the Sobolev norm, and Belomestny et al. (2022) studied the approximation rates for functions in Hölder space in terms of the Hölder norm. Several recent papers have also studied the approximation of derivatives of smooth functions (Duan et al., 2021; Gühring and Raslan, 2021; Belomestny et al., 2022). We will have more detailed discussions on the related works in Section 6.

Table 1 provides a summary of the comparison between our work and the existing results for achieving the same approximation accuracy $\epsilon$ on a function with smoothness index $\beta$ in terms of the needed non-zero parameters in the network. We also summarize the results on whether the neural network approximator has an explicit architecture, where the approximation accuracy holds simultaneously for the target function and its derivative, and whether the approximation results were shown to adapt to the low-dimensional structure of the target function.

In this paper, motivated by the aforementioned estimation problems involving derivatives, we investigate the properties of RePU networks and their derivatives. We show that the partial derivatives of RePU neural networks can be represented by mixed-RePUs activated networks. We derive upper bounds for the complexity of the function class of the derivatives of RePU networks. This is a new result for the complexity of derivatives of RePU networks and is crucial to establish generalization error bounds for a variety of estimation problems involving derivatives, including the score matching estimation and our proposed penalized approach for isotonic regression considered in the present work.

We also derive our approximation results of the RePU network on the smooth functions and their derivatives simultaneously. Our approximation results of the RePU network are based on its representational power on polynomials. We construct the RePU networks with an explicit architecture, which is different from those in the existing literature. The number of hidden layers of our constructed RePU networks only depends on the degree of the target polynomial but is independent of the dimension of input. This construction is new for studying the approximation properties of RePU networks.

We summarize the main contributions of this work as follows.

1. 1.

   We study the basic properties of RePU neural networks and their derivatives. First, we show that partial derivatives of RePU networks can be represented by mixed-RePUs activated networks. We derive upper bounds for the complexity of the function class of partial derivatives of RePU networks in terms of pseudo dimension. Second, we derive novel approximation results for simultaneously approximating $C^{s}$ smooth functions and their derivatives using RePU networks based on a new and efficient construction technique. We show that the approximation can be improved when the data or target function has a low-dimensional structure, which implies that RePU networks can mitigate the curse of dimensionality.

2. 2.

   We study the statistical learning theory of deep score matching estimator (DSME) using RePU networks. We establish non-asymptotic prediction error bounds for DSME under the assumption that the target score is $C^{s}$ smooth. We show that DSME can mitigate the curse of dimensionality if the data has low-dimensional support.

3. 3.

   We propose a penalized deep isotonic regression (PDIR) approach using RePU networks, which encourages the partial derivatives of the estimated regression function to be nonnegative. We establish non-asymptotic excess risk bounds for PDIR under the assumption that the target regression function $f_{0}$ is $C^{s}$ smooth. Moreover, we show that PDIR achieves the minimax optimal rate of convergence for non-parametric regression. We also show that PDIR can mitigate the curse of dimensionality when data concentrates near a low-dimensional manifold. Furthermore, we show that with tuning parameters tending to zero, PDIR is consistent even when the target function is not isotonic.

The rest of the paper is organized as follows. In Section 2 we study basic properties of RePU neural networks. In Section 3 we establish novel approximation error bounds for approximating $C^{s}$ smooth functions and their derivatives using RePU networks. In Section 4 we derive non-asymptotic error bounds for DSME. In Section 5 we propose PDIR and establish non-asymptotic bounds for PDIR. In Section 6 we discuss related works. Concluding remarks are given in Section 7. Results from simulation studies, proofs, and technical details are given in the Supplementary Material.

Neural networks with nonlinear activation functions have proven to be a powerful approach for approximating multi-dimensional functions. One of the most commonly used activation functions is the Rectified linear unit (ReLU), defined as $\sigma_{1}(x)=\max\{x,0\},x\in\mathbb{R}$, due to its attractive properties in computation and optimization. However, since partial derivatives are involved in our objective function (3) and (7), it is not sensible to use networks with piecewise linear activation functions, such as ReLU and Leaky ReLU. Neural networks activated by Sigmoid and Tanh, are smooth and differentiable but have been falling from favor due to their vanishing gradient problems in optimization. In light of these, we are particularly interested in studying the neural networks activated by RePU, which are non-saturated and differentiable.

In Table 2 below we compare RePU with ReLU and Sigmoid networks in several important aspects. ReLU and RePU activation functions are continuous and non-saturated¹¹1An activation function $\sigma$ is saturating if $\lim_{|x|\to\infty}|\nabla\sigma(x)|=0$., which do not have “vanishing gradients” as Sigmodal activations (e.g. Sigmoid, Tanh) in training. RePU and Sigmoid are differentiable and can approximate the gradient of a target function, but ReLU activation is not, especially for estimation involving high-order derivatives of a target function.

We consider the $p$th order Rectified Power units (RePU) activation function for a positive integer $p.$ The RePU activation function, denoted as $\sigma_{p}$, is simply the power of ReLU,

$\displaystyle\sigma_{p}(x)=\left\{\begin{array}[]{lr}x^{p},&x\geq 0\\ 0,&x<0\end{array}\right..$

Note that when $p=0$, the activation function $\sigma_{0}$ is the Heaviside step function; when $p=1$, the activation function $\sigma_{1}$ is the familiar Rectified Linear unit (ReLU); when $p=2,3$, the activation functions $\sigma_{2},\sigma_{3}$ are called rectified quadratic unit (ReQU) and rectified cubic unit (ReCU) respectively. In this work, we focus on the case with $p\geq 2$, implying that the RePU activation function has a continuous $(p-1)$th continuous derivative.

With a RePU activation function, the network will be smooth and differentiable. The architecture of a RePU activated multilayer perceptron can be expressed as a composition of a series of functions

$$f(x)=\mathcal{L}_{\mathcal{D}}\circ\sigma_{p}\circ\mathcal{L}_{\mathcal{D}-1}\circ\sigma_{p}\circ\cdots\circ\sigma_{p}\circ\mathcal{L}_{1}\circ\sigma_{p}\circ\mathcal{L}_{0}(x),\ x\in\mathbb{R}^{d_{0}},$$

where $d_{0}$ is the dimension of the input data, $\sigma_{p}(x)=\{\max(0,x)\}^{p},p\geq 2,$ is a RePU activation function (defined for each component of $x$ if $x$ is a vector), and $\mathcal{L}_{i}(x)=W_{i}x+b_{i},i=0,1,\ldots,\mathcal{D}.$ Here $d_{i}$ is the width (the number of neurons or computational units) of the $i$-th layer, $W_{i}\in\mathbb{R}^{d_{i+1}\times d_{i}}$ is a weight matrix, and $b_{i}\in\mathbb{R}^{d_{i+1}}$ is the bias vector in the $i$-th linear transformation $\mathcal{L}_{i}$. The input data $x$ is the first layer of the neural network and the output is the last layer. Such a network $f$ has $\mathcal{D}$ hidden layers and $(\mathcal{D}+2)$ layers in total. We use a $(\mathcal{D}+2)$-vector $(d_{0},d_{1},\ldots,d_{\mathcal{D}},d_{\mathcal{D}+1})^{\top}$ to describe the width of each layer. The width $\mathcal{W}$ is defined as the maximum width of hidden layers, i.e., $\mathcal{W}=\max\{d_{1},...,d_{\mathcal{D}}\}$; the size $\mathcal{S}$ is defined as the total number of parameters in the network $f$, i.e., $\mathcal{S}=\sum_{i=0}^{\mathcal{D}}\{d_{i+1}\times(d_{i}+1)\}$; the number of neurons $\mathcal{U}$ is defined as the number of computational units in hidden layers, i.e., $\mathcal{U}=\sum_{i=1}^{\mathcal{D}}d_{i}$. Note that the neurons in consecutive layers are connected to each other via weight matrices $W_{i}$, $i=0,1,\ldots,\mathcal{D}$.

We use the notation $\mathcal{F}_{\mathcal{D},\mathcal{W},\mathcal{U},\mathcal{S},\mathcal{B},\mathcal{B}^{\prime}}$ to denote a class of RePU activated multilayer perceptrons $f:\mathbb{R}^{d_{0}}\to\mathbb{R}$ with depth $\mathcal{D}$, width $\mathcal{W}$, number of neurons $\mathcal{U}$, size $\mathcal{S}$ and $f$ satisfying $\|f\|_{\infty}\leq\mathcal{B}$ and $\max_{j=1,\ldots,d}\|\frac{\partial}{\partial x_{j}}f\|_{\infty}\leq\mathcal{B}^{\prime}$ for some $0<\mathcal{B},\mathcal{B}^{\prime}<\infty$, where $\|f\|_{\infty}$ is the sup-norm of a function $f$.

An advantage of RePU networks over piece-wise linear activated networks (e.g. ReLU networks) is that RePU networks are differentiable. Thus RePU networks are useful in many estimation problems involving derivative. To establish the learning theory for these problems, we need to study the properties of derivatives of RePU.

Recall that a $\mathcal{D}$-hidden layer network activated by $p$th order RePU can be expressed by

$$f(x)=\mathcal{L}_{\mathcal{D}}\circ\sigma_{p}\circ\mathcal{L}_{\mathcal{D}-1}\circ\sigma_{p}\circ\cdots\circ\sigma_{p}\circ\mathcal{L}_{1}\circ\sigma_{p}\circ\mathcal{L}_{0}(x),\ x\in\mathbb{R}^{d_{0}}.$$

Let $f_{i}:=\sigma_{p}\circ\mathcal{L}_{i}$ denote the $i$th linear transformation composited with RePU activation for $i=0,1,\ldots,\mathcal{D}-1$ and let $f_{\mathcal{D}}=\mathcal{L}_{\mathcal{D}}$ denotes the linear transformation in the last layer. Then by the chain rule, the gradient of the network can be computed by

$$\nabla f=\left(\prod_{k=0}^{\mathcal{D}-1}\big{[}\nabla f_{\mathcal{D}-k}\circ f_{\mathcal{D}-k-1}\circ\ldots\circ f_{0}\big{]}\right)\nabla f_{0},$$

(8)

where $\nabla$ denotes the gradient operator used in vector calculus. With a differentiable RePU activation $\sigma_{p}$, the gradients $\nabla f_{i}$ in (8) can be exactly computed by $\sigma_{p-1}$ activated layers since $\nabla f_{i}(x)=\nabla[\sigma_{p}\circ\mathcal{L}_{i}(x)]=\nabla\sigma_{p}(W_{i}x+b_{i})=pW_{i}^{\top}\sigma_{p-1}(W_{i}x+b_{i})$. In addition, the $f_{i},i=0,\ldots,\mathcal{D}$ are already RePU-activated layers. Then, the network gradient $\nabla f$ can be represented by a network activated by $\sigma_{p},\sigma_{p-1}$ (and possibly $\sigma_{m}$ for $1\leq m\leq p-2$) according to (8) with a proper architecture. Below, we refer to the neural networks activated by the $\{\sigma_{t}:1\leq t\leq p\}$ as Mixed RePUs activated neural networks, i.e., the activation functions in Mixed RePUs network can be $\sigma_{t}$ for $1\leq t\leq p$, and for different neurons the activation function can be different.

The following theorem shows that the partial derivatives and the gradient $\nabla f_{\phi}$ of a RePU neural network indeed can be represented by a Mixed RePUs network with activation functions $\{\sigma_{t}:1\leq t\leq p\}$.

Theorem 1 shows that for each $j\in\{1,\ldots,d\}$, the partial derivative with respect to the $j$-th argument of the function $f\in\mathcal{F}$ can be exactly computed by a Mixed RePUs network. In addition, by paralleling the networks computing $\frac{\partial}{\partial x_{j}}f,j=1,\ldots,d$, the whole vector of partial derivatives $\nabla f=(\frac{\partial}{\partial x_{1}}f,\ldots,\frac{\partial}{\partial x_{d}}f)$ can be computed by a Mixed RePUs network with depth $3\mathcal{D}+3$, width $6d\mathcal{W}$, number of neurons $13d\mathcal{U}$ and number of parameters $23d\mathcal{S}$.

Let $\mathcal{F}_{j}^{\prime}:=\{{\partial}/{\partial x_{j}}f:f\in\mathcal{F}\}$ be the partial derivatives of the functions in $\mathcal{F}$ with respect to the $j$-th argument. And let $\mathcal{\widetilde{F}}$ denote the class of Mixed RePUs networks in Theorem 1. Then Theorem 1 implies that the class of partial derivative functions is contained in a class of Mixed RePUs networks, i.e., $\mathcal{F}^{\prime}_{j}\subseteq\mathcal{\widetilde{F}}$ for $j=1,\ldots,d$. This further implies the complexity of $\mathcal{F}_{j}^{\prime}$ can be bounded by that of the class of Mixed RePUs networks $\mathcal{\widetilde{F}}$.

The complexity of a function class is a key quantity in the analysis of generalization properties. Lower complexity in general implies a smaller generalization gap. The complexity of a function class can be measured in several ways, including Rademacher complexity, covering number, VC dimension, and Pseudo dimension. These measures depict the complexity of a function class differently but are closely related to each other.

In the following, we develop complexity upper bounds for the class of Mixed RePUs network functions. In particular, these bounds lead to the upper bound for the Pseudo dimension of the function class $\widetilde{\mathcal{F}}$, and that of $\mathcal{F}^{\prime}$.

With Theorem 1 and Lemma 2, we can now obtain an upper bound for the complexity of the class of derivatives of RePU neural networks. This facilitates establishing learning theories for statistical methods involving derivatives.

Due to the symmetry among the arguments of the input of networks in $\mathcal{F}$, the concerned complexities for $\mathcal{F}_{1}^{\prime},\ldots,\mathcal{F}_{d}^{\prime}$ are generally the same. For notational simplicity, we use

$$\mathcal{F}^{\prime}:=\{\frac{\partial}{\partial x_{1}}f:f\in\mathcal{F}\}$$

in the main context to denote the quantities of complexities such as pseudo dimension, e.g., we use ${\rm Pdim}(\mathcal{F}^{\prime})$ instead of ${\rm Pdim}(\mathcal{F}_{j}^{\prime})$ for $j=1,\ldots,d$.

In Theorem 5, to achieve an approximate error $\epsilon$, the RePU neural network should have $O(\epsilon^{-d/s})$ many parameters. The number of parameters grows polynomially in the desired approximation accuracy $\epsilon$ with an exponent $-d/s$ depending on the dimension $d$. In statistical and machine learning tasks, such an approximation result can make the estimation suffer from the curse of dimensionality. In other words, when the dimension $d$ of the input data is large, the convergence rate becomes extremely slow. Fortunately, high-dimensional data often have approximate low-dimensional latent structures in many applications, such as computer vision and natural language processing (Belkin and Niyogi, 2003; Hoffmann et al., 2009; Fefferman et al., 2016). It has been shown that these low-dimensional structures can help mitigate the curse of dimensionality (improve the convergence rate) using ReLU networks (Schmidt-Hieber, 2019; Shen et al., 2020; Jiao et al., 2023; Chen et al., 2022). We consider an assumption of approximate low-dimensional support of data distribution (Jiao et al., 2023), and show that the RePU network can also mitigate the curse of dimensionality under this assumption.

We assume that the high-dimensional data $X$ concentrates in a $\rho$-neighborhood of a low-dimensional manifold. This assumption serves as a relaxation from the stringent requirements imposed by exact manifold assumptions (Chen et al., 2019; Schmidt-Hieber, 2019).

With a well-conditioned manifold $\mathcal{M}$, we show that RePU networks possess the capability to adaptively embed the data into a lower-dimensional space while approximately preserving distances. The dimensionality of the embedded representation, as well as the quality of the embedding in terms of its ability to preserve distances, are contingent upon the properties of the approximate manifold, including its radius $\rho$, condition number $1/\tau$, volume $V$, and geodesic covering regularity $R$. For in-depth definitions of these properties, we direct the interested reader to Baraniuk and Wakin (2009).

When data has a low-dimensional structure, Theorem 8 indicates that the RePU network can approximate $C^{s}$ smooth function up to $l$th order derivatives with an accuracy $\epsilon$ using $O(\epsilon^{d_{\delta}/(s-l)})$ neurons. Here the effective dimension $d_{\delta}$ scales linearly in the intrinsic manifold dimension $d_{\mathcal{M}}$ and logarithmically in the ambient dimension $d$ and the features $1/\tau,V,R$ of the manifold. Compared to Theorem 5, the effective dimensionality in Theorem 8 is $d_{\delta}$ instead of $d$, which could be a significant improvement especially when the ambient dimension of data $d$ is large but the intrinsic dimension $d_{\mathcal{M}}$ is small.

Theorem 8 shows that RePU neural networks are an effective tool for analyzing data that lies in a neighborhood of a low-dimensional manifold, indicating their potential to mitigate the curse of dimensionality. In particular, this property makes them well-suited to scenarios where the ambient dimension of the data is high, but its intrinsic dimension is low. To the best of our knowledge, our Theorem 8 is the first result of the ability of RePU networks to mitigate the curse of dimensionality. A highlight of the comparison between our result and the existing recent results of Li et al. (2019), Li et al. (2020), Duan et al. (2021), Abdeljawad and Grohs (2022) and Belomestny et al. (2022) is given in Table 1.

The development of theory for predicting the performance of score estimator using deep neural networks has been a crucial research area in recent times. Theoretical upper bounds for prediction errors have become increasingly important in understanding the limitations and potential of these models.

We are interested in establishing non-asymptoic error bounds for DSME, which is obtained by minimizing the expected squared distance $\mathbb{E}_{X}\|s(X)-s_{0}(X)\|^{2}_{2}$ over the class of functions $\mathcal{F}$. However, this objective is computationally infeasible because the explicit form of $s_{0}$ is unknown. Under proper conditions, the objective function has an equivalent formulation which is computationally feasible.

Under Assumption 9, the population objective of score matching is equivalent to $J$ given in (3). With a finite sample $S_{n}=\{X_{i}\}_{i=1}^{n}$, the empirical version of $J$ is

$$J_{n}(s)=\frac{1}{n}\sum_{i=1}^{n}\left\{{\rm tr}(\nabla_{x}s(X_{i}))+\frac{1}{2}\|s(X_{i})\|_{2}^{2}\right\}.$$

Then, DSME is defined by

$\displaystyle\hat{s}_{n}:=\arg\min_{s\in\mathcal{F}_{n}}J_{n}(s),$

(9)

which is the empirical risk minimizer over the class of RePU neural networks $\mathcal{F}_{n}$.

Our target is to give upper bounds of the excess risk of $\hat{s}_{n}$, which is defined as

$$J(\hat{s}_{n})-J(s_{0})=\frac{1}{2}\mathbb{E}_{X}\|\hat{s}_{n}(X)-s_{0}(X)\|_{2}^{2}.$$

To obtain an upper bound of $J(\hat{s}_{n})-J(s_{0})$, we decompose it into two parts of error, i.e. stochastic error and approximation error, and then derive upper bounds for them respectively. Let $s_{n}=\arg\min_{s\in\mathcal{F}_{n}}J(s)$, then

	$\displaystyle J(\hat{s}_{n})-J(s_{0})$
	$\displaystyle=\{J(\hat{s}_{n})-J_{n}(\hat{s}_{n})\}+\{J_{n}(\hat{s}_{n})-J_{n}(s_{n})\}+\{J_{n}(s_{n})-J(s_{n})\}+\{J(s_{n})-J(s_{0})\}$
	$\displaystyle\leq\{J(\hat{s}_{n})-J_{n}(\hat{s}_{n})\}+\{J_{n}(s_{n})-J(s_{n})\}+\{J(s_{n})-J(s_{0})\}$
	$\displaystyle\leq 2\sup_{s\in\mathcal{F}_{n}}|J(s)-J_{n}(s)|+\inf_{s\in\mathcal{F}_{n}}\{J(s)-J(s_{0})\},$

where we call $2\sup_{s\in\mathcal{F}_{n}}|J(s)-J_{n}(s)|$ the stochastic error and $\inf_{s\in\mathcal{F}_{n}}\{J(s)-J(s_{0})\}$ the approximation error.