Spectral Regularized Kernel Two-Sample Tests

Given $\mathbb{X}_{N}:=(X_{i})_{i=1}^{N}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}P$, and $\mathbb{Y}_{M}:=(Y_{j})_{j=1}^{M}\stackrel{{\scriptstyle i.i.d.}}{{\sim}}Q$, where $P$ and $Q$ are defined on a measurable space $\mathcal{X}$, the problem of two-sample testing is to test $H_{0}:P=Q$ against $H_{1}:P\neq Q$. This is a classical problem in statistics that has attracted a lot of attention both in the parametric (e.g., $t$-test, $\chi^{2}$-test) and nonparametric (e.g., Kolmogorov-Smirnoff test, Wilcoxon signed-rank test) settings (Lehmann and Romano, 2006). However, many of these tests either rely on strong distributional assumptions or cannot handle non-Euclidean data that naturally arise in many modern applications.

Over the last decade, an approach that has gained a lot of popularity to tackle nonparametric testing problems on general domains is based on the notion of reproducing kernel Hilbert space (RKHS) (Aronszajn, 1950) embedding of probability distributions (Smola et al., 2007, Sriperumbudur et al., 2009, Muandet et al., 2017). Formally, the RKHS embedding of a probability measure $P$ is defined as

$$\mu_{P}=\int_{\mathcal{X}}K(\cdot,x)\,dP(x)\in\mathscr{H},$$

where $K:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$ is the unique reproducing kernel (r.k.) associated with the RKHS $\mathscr{H}$ with $P$ satisfying $\int_{\mathcal{X}}\sqrt{K(x,x)}\,dP(x)<\infty$. If $K$ is characteristic (Sriperumbudur et al., 2010, Sriperumbudur et al., 2011), this embedding induces a metric on the space of probability measures, called the maximum mean discrepancy ($\mathrm{MMD}$) or the kernel distance (Gretton et al., 2012, Gretton et al., 2006), defined as

$$D_{\mathrm{MMD}}(P,Q)=\left\|\mu_{P}-\mu_{Q}\right\|_{\mathscr{H}}.$$

(1.1)

MMD has the following variational representation (Gretton et al., 2012, Sriperumbudur et al., 2010) given by

$$D_{\mathrm{MMD}}(P,Q):=\sup_{f\in\mathscr{H}:\left\|f\right\|_{\mathscr{H}}\leq 1}\int_{\mathcal{X}}f(x)\,d(P-Q)(x),$$

(1.2)

which clearly reduces to (1.1) by using the reproducing property: $f(x)={\left\langle f,K(\cdot,x)\right\rangle}_{\mathscr{H}}$ for all $f\in\mathscr{H}$, $x\in\mathcal{X}$. We refer the interested reader to Sriperumbudur et al. (2010), Sriperumbudur (2016), and Simon-Gabriel and Schölkopf (2018) for more details about $D_{\mathrm{MMD}}$. Gretton et al. (2012) proposed a test based on the asymptotic null distribution of the $U$-statistic estimator of $D^{2}_{\mathrm{MMD}}(P,Q)$, defined as

	$\displaystyle\hat{D}_{\mathrm{MMD}}^{2}(X,Y)$	$\displaystyle=\frac{1}{N(N-1)}\sum_{i\neq j}K(X_{i},X_{j})+\frac{1}{M(M-1)}\sum_{i\neq j}K(Y_{i},Y_{j})$
		$\displaystyle\qquad\qquad-\frac{2}{NM}\sum_{i,j}K(X_{i},Y_{j}),$

and showed it to be consistent. Since the asymptotic null distribution does not have a simple closed form—the distribution is that of an infinite sum of weighted chi-squared random variables with the weights being the eigenvalues of an integral operator associated with the kernel $K$ w.r.t. the distribution $P$—, several approximate versions of this test have been investigated and are shown to be asymptotically consistent (e.g., see Gretton et al., 2012 and Gretton et al., 2009). Recently, Li and Yuan (2019) and Schrab et al. (2021) showed these tests based on $\hat{D}^{2}_{\text{MMD}}$ to be not optimal in the minimax sense but modified them to achieve a minimax optimal test by using translation-invariant kernels on $\mathcal{X}=\mathbb{R}^{d}$. However, since the power of these kernel methods lies in handling more general spaces and not just $\mathbb{R}^{d}$, the main goal of this paper is to construct minimax optimal kernel two-sample tests on general domains.

Before introducing our contributions, first, we will introduce the minimax framework pioneered by Burnashev (1979) and Ingster (1987, 1993) to study the optimality of tests, which is essential to understanding our contributions and their connection to the results of Li and Yuan (2019) and Schrab et al. (2021). Let $\phi(\mathbb{X}_{N},\mathbb{Y}_{M})$ be any test that rejects $H_{0}$ when $\phi=1$ and fails to reject $H_{0}$ when $\phi=0$. Denote the class of all such asymptotic (resp. exact) $\alpha$-level tests to be $\Phi_{\alpha}$ (resp. $\Phi_{N,M,\alpha}$). Let $\mathcal{C}$ be a set of probability measures on $\mathcal{X}$. The Type-II error of a test $\phi\in\Phi_{\alpha}$ (resp. $\in\Phi_{N,M,\alpha}$) w.r.t. $\mathcal{P}_{\Delta}$ is defined as $R_{\Delta}(\phi)=\sup_{(P,Q)\in\mathcal{P}_{\Delta}}\mathbb{E}_{P^{N}\times Q^{M}}(1-\phi),$ where

$$\mathcal{P}_{\Delta}:=\left\{(P,Q)\in\mathcal{C}^{2}:\rho^{2}(P,Q)\geq\Delta\right\},$$

is the class of $\Delta$-separated alternatives in probability metric $\rho$, with $\Delta$ being referred to as the separation boundary or contiguity radius. Of course, the interest is in letting $\Delta\rightarrow 0$ as $M,N\rightarrow\infty$ (i.e., shrinking alternatives) and analyzing $R_{\Delta}$ for a given test, $\phi$, i.e., whether $R_{\Delta}(\phi)\rightarrow 0$. In the asymptotic setting, the minimax separation or critical radius $\Delta^{*}$ is the fastest possible order at which $\Delta\rightarrow 0$ such that $\lim\inf_{N,M\rightarrow\infty}\inf_{\phi\in\Phi_{\alpha}}R_{\Delta^{*}}(\phi)\rightarrow 0$, i.e., for any $\Delta$ such that $\Delta/\Delta^{*}\rightarrow\infty$, there is no test $\phi\in\Phi_{\alpha}$ that is consistent over $\mathcal{P}_{\Delta}$. A test is asymptotically minimax optimal if it is consistent over $\mathcal{P}_{\Delta}$ with $\Delta\asymp\Delta^{*}$. On the other hand, in the non-asymptotic setting, the minimax separation $\Delta^{*}$ is defined as the minimum possible separation, $\Delta$ such that $\inf_{\phi\in\Phi_{N,M,\alpha}}R_{\Delta}(\phi)\leq\delta$, for $0<\delta<1-\alpha$. A test $\phi\in\Phi_{N,M,\alpha}$ is called minimax optimal if $R_{\Delta}(\phi)\leq\delta$ for some $\Delta\asymp\Delta^{*}$. In other words, there is no other $\alpha$-level test that can achieve the same power with a better separation boundary.

In the context of the above notation and terminology, Li and Yuan (2019) considered distributions with densities (w.r.t. the Lebesgue measure) belonging to

$$\mathcal{C}=\left\{f:\mathbb{R}^{d}\rightarrow\mathbb{R}\,:\,f\,\,\text{a.s. continuous and}\,\,\|f\|_{W^{s,2}_{d}}\leq M\right\},$$

(1.3)

where $\|f\|^{2}_{W^{s,2}_{d}}=\int(1+\|x\|^{2}_{2})^{s}|\hat{f}(x)|^{2}\,dx$ with $\hat{f}$ being the Fourier transform of $f$, $\rho(P,Q)=\|p-q\|_{L^{2}}$ with $p$ and $q$ being the densities of $P$ and $Q$, respectively, and showed the minimax separation to be $\Delta^{*}\asymp(N+M)^{-4s/(4s+d)}$. Furthermore, they chose $K$ to be a Gaussian kernel on $\mathbb{R}^{d}$, i.e., $K(x,y)=\exp(-\|x-y\|^{2}_{2}/h)$ in $\hat{D}^{2}_{\text{MMD}}$ with $h\rightarrow 0$ at an appropriate rate as $N,M\rightarrow\infty$ (reminiscent of kernel density estimators) in contrast to fixed $h$ in Gretton et al. (2012), and showed the resultant test to be asymptotically minimax optimal w.r.t. $\mathcal{P}_{\Delta}$ based on (1.3) and $\|\cdot\|_{L^{2}}$. Schrab et al. (2021) extended this result to translation-invariant kernels (particularly, as the product of one-dimensional translation-invariant kernels) on $\mathcal{X}=\mathbb{R}^{d}$ with a shrinking bandwidth and showed the resulting test to be minimax optimal even in the non-asymptotic setting. While these results are interesting, the analysis holds only for $\mathbb{R}^{d}$ as the kernels are chosen to be translation invariant on $\mathbb{R}^{d}$, thereby limiting the power of the kernel approach.

In this paper, we employ an operator theoretic perspective to understand the limitation of $D^{2}_{\text{MMD}}$ and propose a regularized statistic that mitigates these issues without requiring $\mathcal{X}=\mathbb{R}^{d}$. In fact, the construction of the regularized statistic naturally gives rise to a certain $\mathcal{P}_{\Delta}$ which is briefly described below. To this end, define $R:=\frac{P+Q}{2}$ and $u:=\frac{dP}{dR}-1$ which is well defined as $P\ll R$. It can be shown that $D^{2}_{\text{MMD}}(P,Q)=4\langle\mathcal{T}u,u\rangle_{L^{2}(R)}$ where $\mathcal{T}:L^{2}(R)\rightarrow L^{2}(R)$ is an integral operator defined by $K$ (see Section 3 for details), which is in fact a self-adjoint positive trace-class operator if $K$ is bounded. Therefore, $D^{2}_{\text{MMD}}(P,Q)=\sum_{i}\lambda_{i}\langle u,\tilde{\phi}_{i}\rangle^{2}_{L^{2}(R)}$ where $(\lambda_{i},\tilde{\phi}_{i})_{i}$ are the eigenvalues and eigenfunctions of $\mathcal{T}$. Since $\mathcal{T}$ is trace-class, we have $\lambda_{i}\rightarrow 0$ as $i\rightarrow\infty$, which implies that the Fourier coefficients of $u$, i.e., $\langle u,\tilde{\phi}_{i}\rangle_{L^{2}(R)}$, for large $i$, are down-weighted by $\lambda_{i}$. In other words, $D^{2}_{\text{MMD}}$ is not powerful enough to distinguish between $P$ and $Q$ if they differ in the high-frequency components of $u$, i.e., $\langle u,\tilde{\phi}_{i}\rangle_{L^{2}(R)}$ for large $i$. On the other hand,

$$\|u\|^{2}_{L^{2}(R)}=\sum_{i}\langle u,\tilde{\phi}_{i}\rangle^{2}_{L^{2}(R)}=\chi^{2}\left(P\left\|\right.\frac{P+Q}{2}\right)=\frac{1}{2}\int_{\mathcal{X}}\frac{(dP-dQ)^{2}}{d(P+Q)}=:\underline{\rho}^{2}(P,Q)$$

does not suffer from any such issue, with $\underline{\rho}$ being a probability metric that is topologically equivalent to the Hellinger distance (see Lemma A.18 and Le Cam, 1986, p. 47). With this motivation, we consider the following modification to $D^{2}_{\text{MMD}}$:

$$\eta_{\lambda}(P,Q)=4\left\langle\mathcal{T}g_{\lambda}(\mathcal{T})u,u\right\rangle_{L^{2}(R)},$$

where $g_{\lambda}:(0,\infty)\rightarrow(0,\infty)$, called the spectral regularizer (Engl et al., 1996) is such that $\lim_{\lambda\rightarrow 0}xg_{\lambda}(x)\asymp 1$ as $\lambda\rightarrow 0$ (a popular example is the Tikhonov regularizer, $g_{\lambda}(x)=\frac{1}{x+\lambda}$), i.e., $\mathcal{T}g_{\lambda}(\mathcal{T})\approx\boldsymbol{I}$, the identity operator—refer to (4.1) for the definition of $g_{\lambda}(\mathcal{T})$. In fact, in Section 4, we show $\eta_{\lambda}(P,Q)$ to be equivalent to $\|u\|^{2}_{L^{2}(R)}$, i.e., $\|u\|^{2}_{L^{2}(R)}\lesssim\eta_{\lambda}(P,Q)\lesssim\|u\|^{2}_{L^{2}(R)}$ if $u\in\text{Ran}(\mathcal{T}^{\theta}),\,\theta>0$ and $\|u\|^{2}_{L^{2}(R)}\gtrsim\lambda^{2\theta}$, where $\text{Ran}(A)$ denotes the range space of an operator $A$, $\theta$ is the smoothness index (large $\theta$ corresponds to “smooth” $u$), and $\mathcal{T}^{\theta}$ is defined by choosing $g_{\lambda}(x)=x^{\theta},\,x\geq 0$ in (4.1). This naturally leads to the class of $\Delta$-separated alternatives,

$$\mathcal{P}:=\mathcal{P}_{\theta,\Delta}:=\left\{(P,Q):\frac{dP}{dR}-1\in\text{Ran}(\mathcal{T}^{\theta}),\,\,\underline{\rho}^{2}(P,Q)\geq\Delta\right\},$$

(1.4)

for $\theta>0$, where $\text{Ran}(\mathcal{T}^{\theta}),\,\theta\in(0,\frac{1}{2}]$ can be interpreted as an interpolation space obtained by the real interpolation of $\mathscr{H}$ and $L^{2}(R)$ at scale $\theta$ (Steinwart and Scovel, 2012, Theorem 4.6)—note that the real interpolation of Sobolev spaces and $L^{2}(\mathbb{R}^{d})$ yields Besov spaces (Adams and Fournier, 2003, p. 230). To compare the class in (1.4) to that obtained using (1.3) with $\rho(\cdot,\cdot)=\|\cdot-\cdot\|_{L^{2}(\mathbb{R}^{d})}$, note that the smoothness in (1.4) is determined through $\text{Ran}(\mathcal{T}^{\theta})$ instead of the Sobolev smoothness where the latter is tied to translation-invariant kernels on $\mathbb{R}^{d}$. Since we work with general domains, the smoothness is defined through the interaction between $K$ and the probability measures in terms of the behavior of the integral operator, $\mathcal{T}$. In addition, as $\lambda\rightarrow 0$, $\eta_{\lambda}(P,Q)\rightarrow\underline{\rho}^{2}(P,Q)$, while $D^{2}_{\text{MMD}}(P,Q)\rightarrow\|p-q\|^{2}_{2}$ as $h\rightarrow 0$, where $D^{2}_{\text{MMD}}$ is defined through a translation invariant kernel on $\mathbb{R}^{d}$ with bandwidth $h>0$. Hence, we argue that (1.4) is a natural class of alternatives to investigate the performance of $D^{2}_{\text{MMD}}$ and $\eta_{\lambda}$. In fact, recently, Balasubramanian et al. (2021) considered an alternative class similar to (1.4) to study goodness-of-fit tests using $D^{2}_{\text{MMD}}$.

For a topological space $\mathcal{X}$, $L^{r}(\mathcal{X},\mu)$ denotes the Banach space of $r$-power $(r\geq 1)$ $\mu$-integrable function, where $\mu$ is a finite non-negative Borel measure on $\mathcal{X}$. For $f\in L^{r}(\mathcal{X},\mu)=:L^{r}(\mu)$, $\left\|f\right\|_{L^{r}(\mu)}:=(\int_{\mathcal{X}}|f|^{r}\,d\mu)^{1/r}$ denotes the $L^{r}$-norm of $f$. $\mu^{n}:=\mu\times\stackrel{{\scriptstyle n}}{{...}}\times\mu$ is the $n$-fold product measure. $\mathscr{H}$ denotes a reproducing kernel Hilbert space with a reproducing kernel $K:\mathcal{X}\times\mathcal{X}\to\mathbb{R}$. $[f]_{\sim}$ denotes the equivalence class of the function $f$, that is the collection of functions $g\in L^{r}(\mathcal{X},\mu)$ such that $\left\|f-g\right\|_{L^{r}(\mu)}=0$. For two measures $P$ and $Q$, $P\ll Q$ denotes that $P$ is dominated by $Q$ which means, if $Q(A)=0$ for some measurable set $A$, then $P(A)=0$.

Let $H_{1}$ and $H_{2}$ be abstract Hilbert spaces. $\mathcal{L}(H_{1},H_{2})$ denotes the space of bounded linear operators from $H_{1}$ to $H_{2}$. For $S\in\mathcal{L}(H_{1},H_{2})$, $S^{*}$ denotes the adjoint of $S$. $S\in\mathcal{L}(H):=\mathcal{L}(H,H)$ is called self-adjoint if $S^{*}=S$. For $S\in\mathcal{L}(H)$, $\text{Tr}(S)$, $\left\|S\right\|_{\mathcal{L}^{2}(H)}$, and $\left\|S\right\|_{\mathcal{L}^{\infty}(H)}$ denote the trace, Hilbert-Schmidt and operator norms of $S$, respectively. For $x,y\in H$, $x\otimes_{H}y$ is an element of the tensor product space of $H\otimes H$ which can also be seen as an operator from $H\to H$ as $(x\otimes_{H}y)z=x{\left\langle y,z\right\rangle}_{H}$ for any $z\in H$.

For constants $a$ and $b$, $a\lesssim b$ (resp. $a\gtrsim b$) denotes that there exists a positive constant $c$ (resp. $c^{\prime}$) such that $a\leq cb$ (resp. $a\geq c^{\prime}b)$. $a\asymp b$ denotes that there exists positive constants $c$ and $c^{\prime}$ such that $cb\leq a\leq c^{\prime}b$. We denote $[\ell]$ for $\{1,\ldots,\ell\}$.

In this section, we establish the non-optimality of the test based on $D^{2}_{\text{MMD}}$. First, we make the following assumption throughout the paper.
$(A_{0})$ $(\mathcal{X},\mathcal{B})$ is a second countable (i.e., completely separable) space endowed with Borel $\sigma$-algebra $\mathcal{B}$. $(\mathscr{H},K)$ is an RKHS of real-valued functions on $\mathcal{X}$ with a continuous reproducing kernel $K$ satisfying $\sup_{x}K(x,x)\leq\kappa.$
The continuity of $K$ ensures that $K(\cdot,x):\mathcal{X}\to\mathscr{H}$ is Bochner-measurable for all $x\in\mathcal{X}$, which along with the boundedness of $K$ ensures that $\mu_{P}$ and $\mu_{Q}$ are well-defined (Dinculeanu, 2000). Also the separability of $\mathcal{X}$ along with the continuity of $K$ ensures that $\mathscr{H}$ is separable (Steinwart and Christmann, 2008, Lemma 4.33). Therefore,

	$\displaystyle D^{2}_{\mathrm{MMD}}(P,Q)$	$\displaystyle={\left\langle\int_{\mathcal{X}}K(\cdot,x)\,d(P-Q)(x),\int_{\mathcal{X}}K(\cdot,x)\,d(P-Q)(x)\right\rangle}_{\mathscr{H}}$
		$\displaystyle=4{\left\langle\int_{\mathcal{X}}K(\cdot,x)u(x)\,dR(x),\int_{\mathcal{X}}K(\cdot,x)u(x)\,dR(x)\right\rangle}_{\mathscr{H}},$		(3.1)

where $R=\frac{P+Q}{2}$ and $u=\frac{dP}{dR}-1$. Define $\mathfrak{I}:\mathscr{H}\to L^{2}(R)$, $f\mapsto[f-\mathbb{E}_{R}f]_{\sim}$, which is usually referred in the literature as the inclusion operator (e.g., see Steinwart and Christmann, 2008, Theorem 4.26), where $\mathbb{E}_{R}f=\int_{\mathcal{X}}f(x)\,dR(x)$. It can be shown (Sriperumbudur and Sterge, 2022, Proposition C.2) that $\mathfrak{I}^{*}:L^{2}(R)\to\mathscr{H}$, $f\mapsto\int K(\cdot,x)f(x)\,dR(x)-\mu_{R}\mathbb{E}_{R}f$. Define $\mathcal{T}:=\mathfrak{I}\mathfrak{I}^{*}:L^{2}(R)\to L^{2}(R)$. It can be shown (Sriperumbudur and Sterge, 2022, Proposition C.2) that $\mathcal{T}=\Upsilon-(1\otimes_{L^{2}(R)}1)\Upsilon-\Upsilon(1\otimes_{L^{2}(R)}1)+(1\otimes_{L^{2}(R)}1)\Upsilon(1\otimes_{L^{2}(R)}1)$, where $\Upsilon:L^{2}(R)\to L^{2}(R)$, $f\mapsto\int K(\cdot,x)f(x)\,dR(x)$. Since $K$ is bounded, it is easy to verify that $\mathcal{T}$ is a trace class operator, and thus compact. Also, it is self-adjoint and positive, thus spectral theorem (Reed and Simon, 1980, Theorems VI.16, VI.17) yields that

$$\mathcal{T}=\sum_{i\in I}\lambda_{i}\tilde{\phi_{i}}\otimes_{L^{2}(R)}\tilde{\phi_{i}},$$

where $(\lambda_{i})_{i}\subset\mathbb{R}^{+}$ are the eigenvalues and $(\tilde{\phi}_{i})_{i}$ are the orthonormal system of eigenfunctions (strictly speaking classes of eigenfunctions) of $\mathcal{T}$ that span $\overline{\text{Ran}(\mathcal{T})}$ with the index set $I$ being either countable in which case $\lambda_{i}\to 0$ or finite. In this paper, we assume that the set $I$ is countable, i.e., infinitely many eigenvalues. Note that $\tilde{\phi_{i}}$ represents an equivalence class in $L^{2}(R)$. By defining $\phi_{i}:=\frac{\mathfrak{I}^{*}\tilde{\phi_{i}}}{\lambda_{i}}$, it is clear that $\mathfrak{I}\phi_{i}=[\phi_{i}-\mathbb{E}_{R}\phi_{i}]_{\sim}=\tilde{\phi_{i}}$ and $\phi_{i}\in\mathscr{H}$. Throughout the paper, $\phi_{i}$ refers to this definition. Using these definitions, we can see that

$\displaystyle D^{2}_{\mathrm{MMD}}(P,Q)$

$\displaystyle=4{\left\langle\mathfrak{I}^{*}u,\mathfrak{I}^{*}u\right\rangle}_{\mathscr{H}}=4{\left\langle\mathcal{T}u,u\right\rangle}_{L^{2}(R)}=4\sum_{i\geq 1}\lambda_{i}\langle u,\tilde{\phi_{i}}\rangle^{2}_{L^{2}(R)}.$

(3.2)

Suppose $u\in\text{span}\{\tilde{\phi}_{i}:i\in I\}$. Then $\sum_{i\geq 1}\langle u,\tilde{\phi_{i}}\rangle^{2}_{L^{2}(R)}=\left\|u\right\|_{L^{2}(R)}^{2}\stackrel{{\scriptstyle(*)}}{{=}}\underline{\rho}^{2}(P,Q),$ where $\underline{\rho}^{2}(P,Q):=\frac{1}{2}\int\frac{(dP-dQ)^{2}}{dP+dQ}$ and $(*)$ follows from Lemma A.18 by noting that $\|u\|^{2}_{L^{2}(R)}=\chi^{2}(P||R)$. As mentioned in Section 1, $D_{\mathrm{MMD}}$ might not capture the difference between between $P$ and $Q$ if they differ in the higher Fourier coefficients of $u$, i.e., $\langle u,\tilde{\phi}_{i}\rangle_{L^{2}(R)}$ for large $i$.

The following result shows that the test based on $\hat{D}_{\mathrm{MMD}}^{2}$ cannot achieve a separation boundary of order better than $(N+M)^{\frac{-2\theta}{2\theta+1}}$.

The following result provides general conditions on the minimax separation rate w.r.t. $\mathcal{P}$, which together with Corollaries 3.3 and 3.4 demonstrates the non-optimality of the MMD tests presented in Theorem 3.1.

To address the limitation of the MMD test, in this section, we propose a spectral regularized version of the MMD test and show it to be minimax optimal w.r.t. $\mathcal{P}$. To this end, we define the spectral regularized discrepancy as

$$\eta_{\lambda}(P,Q):=4{\left\langle\mathcal{T}g_{\lambda}(\mathcal{T})u,u\right\rangle}_{L^{2}(R)},$$

where the spectral regularizer, $g_{\lambda}:(0,\infty)\rightarrow(0,\infty)$ satisfies $\lim_{\lambda\rightarrow 0}xg_{\lambda}(x)\asymp 1$ (more concrete assumptions on $g_{\lambda}$ will be introduced later). By functional calculus, we define $g_{\lambda}$ applied to any compact, self-adjoint operator $\mathcal{B}$ defined on a separable Hilbert space, $H$ as

$$g_{\lambda}(\mathcal{B}):=\sum_{i\geq 1}g_{\lambda}(\tau_{i})(\psi_{i}\otimes_{H}\psi_{i})+g_{\lambda}(0)\left(\boldsymbol{I}-\sum_{i\geq 1}\psi_{i}\otimes_{H}\psi_{i}\right),$$

(4.1)

where $\mathcal{B}$ has the spectral representation, $\mathcal{B}=\sum_{i}\tau_{i}\psi_{i}\otimes_{H}\psi_{i}$ with $(\tau_{i},\psi_{i})_{i}$ being the eigenvalues and eigenfunctions of $\mathcal{B}$. A popular example of $g_{\lambda}$ is $g_{\lambda}(x)=\frac{1}{x+\lambda}$, yielding $g_{\lambda}(\mathcal{B})=(\mathcal{B}+\lambda\boldsymbol{I})^{-1}$, which is well known as the Tikhonov regularizer. We will later provide more examples of spectral regularizers that satisfy additional assumptions.