Nonparametric inference on non-negative dissimilarity measures at the boundary of the parameter space

Let $Z_{1},\ldots,Z_{n}$ be i.i.d. random vectors, generated from an unknown probability distribution $P_{0}$. We make few assumptions about $P_{0}$ and only require that it belongs to a flexible nonparametric model $\mathcal{M}$, which is essentially unrestricted, aside from mild regularity conditions. For a given probability distribution $P$ in $\mathcal{M}$, let $\theta_{P}$ be a function-valued summary of interest with domain $\mathcal{O}\subseteq\mathbb{R}^{d}$ for a positive integer $d$ and range $\mathcal{K}\subseteq\mathbb{R}$. We denote by $\theta_{0}:=\theta_{P_{0}}$ the evaluation of this summary at $P_{0}$.

Suppose that $\theta_{P}$ is known to belong to a, possibly infinite-dimensional, function class $\Theta$. For a given distribution $P$, we define a real-valued functional $G_{P}:\Theta\to\mathbb{R}$ that satisfies

The functional $G_{P}$ measures the goodness-of-fit of any function $\theta\in\Theta$ – larger values of $G_{P}(\theta)$ indicate that $\theta$ and $\theta_{P}$ are farther away from one another, in a sense. Throughout this paper, we use the shorthand notation $G_{0}:=G_{P_{0}}$ to denote the value of the goodness-of-fit measure at $P_{0}$.

Let $\Theta^{*}\subset\Theta$ be a subclass of $\Theta$, and let $\theta_{P}^{*}$ be a function that satisfies

In essence, $\theta^{*}_{P}$ is the closest function to $\theta_{P}$ among all functions in the subclass $\Theta^{*}$. We define as our target parameter the difference between the goodness-of-fit of $\theta_{P}$ and $\theta^{*}_{P}$,

and we again use the shorthand notation $\Psi_{0}:=\Psi_{P_{0}}$. Throughout this manuscript, we refer to $\Psi_{0}$ as the improvement in fit because it represents the improvement in the goodness-of-fit attained by using the full function class instead of the reduced class.

Because $\Theta^{*}$ is contained within $\Theta$, it can be seen that $\Psi_{P}$ is a non-negative statistical functional, and $\Psi_{P}$ is only equal to zero when $\theta_{P}$ provides no improvement in fit compared with $\theta^{*}_{P}$. In many applications, a problem of central importance is to determine whether $\theta^{*}_{P}$ is inferior to $\theta_{P}$ in terms of goodness-of-fit. Letting $\Psi_{0}:=\Psi_{P_{0}}$, we are interested in performing a test of the null hypothesis

Additionally, because statistical functionals that have the representation in (2) have scientifically meaningful interpretations in some contexts, estimation of $\Psi_{0}$ and confidence interval construction are also of practical interest. Our paper provides a general framework for estimation, testing, and confidence interval construction for statistical functionals of this form.

In what follows, we introduce some working examples. As a first example, we discuss statistical inference in nonparametric regression models, and second, we discuss a nonparametric approach for assessing dependence between a pair of random variables. We then describe a simple way to define a goodness-of-fit measure for any function-valued parameter.

We are interested in studying the conditional mean of the outcome given the predictors, defined as $\theta_{P}:(w,x)\mapsto E_{P}[Y|X=x,W=w]$. It is well-known that the conditional mean can be characterized as the minimizer of the expected squared error loss over $\Theta$, if $\Theta$ is sufficiently large. That is, defining the goodness-of-fit measure

the conditional mean satisfies $G_{P}(\theta_{P})=\inf_{\theta\in\Theta}G_{P}(\theta)$.

Consider now the set of candidate prediction functions that do not depend on $X$, which we write as

When $\Theta$ is large, any minimizer $\theta^{*}_{P}$ of the expected squared error loss over $\Theta^{*}$ is almost everywhere equal to the conditional mean of $Y$ given $W$. We are often interested in determining whether $X$ is an important set of predictors in the sense that it does not need to be included in a prediction function in order for optimal squared error loss to be achieved. If $X$ is not important in this sense, the conditional mean of $Y$ given $X$ and $W$ does not depend on $X$, and the difference in the expected squared error loss $\Psi_{0}=G_{0}(\theta^{*}_{0})-G_{0}(\theta_{0})$ is zero. Otherwise, $\Psi_{0}$ is positive. Thus, assessing variable importance can be framed a statistical inference problem of the type described in Section 2.

Many recent works have studied inference on variable importance estimands of a similar form to that we describe above (see, e.g., Williamson et al., 2021a, b; Verdinelli and Wasserman, 2021; Zhang and Janson, 2020). These works all encounter difficulties with constructing estimators for their original target estimand that achieve parametric rate convergence under the null. To the best of our knowledge, there is currently no solution available to this problem.

Let $\Theta$ be a collection of candidate values for the log density function, and assume that $\Theta$ is large enough to contain $\theta_{0}$, the log density under $P_{0}$. The density function can be represented as a minimizer of the expected cross-entropy loss. Therefore, defining the goodness-of-fit measure

the joint density satisfies (1).

We now define $\Theta^{*}$ as the class of candidate log density functions for which the joint density can be expressed as the product of two marginal density functions – that is,

Any minimizer $\theta^{*}_{P}$ of $G_{P}$ over $\Theta^{*}$ is almost everywhere equal to the product of the marginal densities of $X$ and $Y$ under $P$. Therefore, $\Psi_{0}:=G_{0}(\theta^{*}_{0})-G_{0}(\theta_{0})$ is zero if $X$ and $Y$ are independent, and $\Psi_{0}$ is otherwise positive. One can assess dependence between $X$ and $Y$ by performing inference on $\Psi_{0}$, so similar to the previous example, this problem falls within our framework.

The measure of dependence $\Psi_{0}$ we have defined here is commonly referred to as the mutual information and has been a widely-studied measure of stochastic dependence (see, e.g., Paninski, 2003; Steuer et al., 2002). We are not aware of an existing nonparametric estimator for the mutual information that achieves parametric rate convergence under the null of independence. This appears to be a longstanding open problem.

Suppose one is interested in assessing whether a given function-valued parameter $\theta_{P}$ is equal to a fixed and known function $\theta^{*}$. For a measure $\nu$ on $\mathcal{O}$, one can define as a goodness-of-fit measure an integrated squared difference between $\theta_{P}$ and $\theta^{*}$:

Because $G_{P}$ is non-negative, and $G_{P}(\theta_{P})=0$, it is easy to see that $G_{P}$ is minimized by $\theta_{P}$.

One might wish to perform inference on the quantity

Clearly, $\Psi_{0}$ is equal to zero only when $\theta_{0}$ is equal to $\theta^{*}$ almost everywhere $\nu$. Estimands of this form can be of interest when one wishes to construct an omnibus test of the hypothesis that $\theta_{0}=\theta^{*}$. The framework we develop in this paper can be applied in this setting as well, and so, methodology for inference on the improvement in fit can be seen as generally useful for performing inference on function-valued parameters.

Our proposal requires us to construct an estimator for $\{G_{0,f}(\beta):f\in\mathcal{F},\beta\in\mathcal{B}\}$ that enables us to perform inference uniformly along the collection of parametric sub-models. In this sub-section, we first outline a set of sufficient conditions under which an estimator has asymptotic properties that facilitate uniform inference. We then describe a strategy for constructing an estimator that satisfies these conditions.

We begin by providing assumptions upon which our first main theoretical result relies. We consider two types of assumptions. The first type (A) is a set of determinsitic conditions on the goodness-of-fit functional and the underlying probabilty distribution, whereas the second set of assumptions (B) is stochastic in nature and describes conditions that our estimator $\{\tilde{G}_{n,f}(\beta):f\in\mathcal{F},\beta\in\mathcal{B}\}$ must satisfy.

• Assumption A1: For any $f\in\mathcal{F}$ and any $\beta\in\mathcal{B}$, $G_{P,f}(\beta)$ is pathwise differentiable in a nonparametric model, and its nonparametric efficient influence function is given by $\phi_{P,f}(\cdot;\beta):\mathcal{Z}\to\mathbb{R}$.

• Assumption A2: $G_{P,f}$ and $\phi_{P,f}$ are twice differentiable in $\beta$ for each $f$ in $\mathcal{F}$, and the derivatives are given by

  	$\displaystyle G^{\prime}_{P,f}(\beta):=\frac{d}{d\beta}G_{P,f}(\beta),\quad G^{\prime\prime}_{P,f}(\beta):=\frac{d^{2}}{d\beta^{2}}G_{P,f}(\beta),$
  	$\displaystyle\phi^{\prime}_{P,f}(\cdot;\beta):=\frac{d}{d\beta}\phi_{P,f}(\cdot;\beta),\quad\phi^{\prime\prime}_{P,f}(\beta):=\frac{d^{2}}{d\beta^{2}}\phi_{P,f}(\cdot;\beta).$

• Assumption A3: There exist positive constants $C_{1},C_{2}>0$ such that, for any $\{\beta_{f}:\in\mathcal{F}\}$, $\sup_{f\in\mathcal{F}}G_{0,f}(\beta_{f})-G_{0,f}(\beta_{0,f})<C_{1}$ implies that

  $\displaystyle\sup_{f\in\mathcal{F}}(\beta_{0,f}-\beta_{f})^{2}\leq C_{2}\left\{\sup_{f\in\mathcal{F}}G_{0,f}(\beta_{f})-G_{0,f}(\beta_{0,f})\right\}.$

• Assumption A4: For each $f\in\mathcal{F}$, $G^{\prime}_{0,f}(\beta_{0,f})=0$. Additionally, $G^{\prime\prime}_{0,f}$ is bounded above zero in a neighborhood of $\beta_{0,f}$, uniformly in $\mathcal{F}$. That is, $\inf_{f\in\mathcal{F}}G^{\prime\prime}_{0,f}(\beta_{f})$ is positive whenever $\sup_{f\in\mathcal{F}}|\beta_{f}-\beta_{0,f}|$ is small.

• Assumption A5: Both the function classes $\left\{\phi_{P_{0},f}(\cdot;\beta):f\in\mathcal{F},\beta\in\mathcal{B}\right\}$ and $\left\{\phi^{\prime}_{P_{0},f}(\cdot;\beta_{0,f}):f\in\mathcal{F}\right\}$ are $P_{0}$-Donsker.

Assumption A1 requires that the goodness-of-fit is pathwise differentiable, which as noted in Section 3, enables us to construct $n^{1/2}$-consistent estimators. When $G_{P,f}$ is pathwise differentiable estimand, its efficient influence function is guaranteed to exist, and knowledge of the efficient influence function is often needed for constructing efficient estimators and studying their asymptotic properties in nonparametric models. We note that because we assume $G_{P,f_{1}}(0)=G_{P,f_{2}}(0)$ for any $f_{1},f_{2}$ (recall we assume (7) holds), the efficient influence functions $\phi_{P,f_{1}}(\cdot;0)$ and $\phi_{P,f_{2}}(\cdot;0)$ are also equal. Assumptions A2 and A3 state that the goodness-of-fit must be smooth and convex along each of the parametric sub-models. Assumption A4 requires that $\mathcal{B}$ is large enough to contain the global optimizer of $G_{0,f}$ over $\mathbb{R}$, and the goodness-of-fit satisfies some additional smoothness constraints in a neighborhood of the optimizer. Assumption A5 states that, while $\mathcal{F}$ may be specified as a large nonparametric function class, it must satisfy some mild complexity constraints.

• Assumption B1: For any $f_{1},f_{2}\in\mathcal{F}$ with $f_{1}\neq f_{2}$ we have that $\tilde{G}_{n,f_{1}}(0)=\tilde{G}_{n,f_{2}}(0)$.

• Assumption B2: $\tilde{G}_{n,f}$ is an asymptotically linear estimator for $G_{0,f}$ in the sense that the remainder $\tilde{r}_{n,f}(\beta):=\left\{\tilde{G}_{n,f}(\beta)-G_{0,f}(\beta)\right\}-\frac{1}{n}\sum_{i=1}^{n}\phi_{0,f}(Z_{i};\beta)$ satisfies

  $\displaystyle\sup_{f\in\mathcal{F},\beta\in\mathcal{B}}|\tilde{r}_{n,f}(\beta)|=o_{P}(n^{-1/2}).$

• Assumption B3: The derivative of $\tilde{G}_{n,f}$ exists and is given by $\tilde{G}^{\prime}_{n,f}(\beta)=\frac{d}{d\beta}\tilde{G}_{n,f}(\beta)$. Moreover, letting $\tilde{r}^{\prime}_{n,f}(\beta)=\frac{d}{d\beta}r_{n,f}(\beta)$, we have

  $\displaystyle\sup_{f\in\mathcal{F},\beta\in\mathcal{B}}|\tilde{r}^{\prime}_{n,f}(\beta)|=o_{P}(n^{-1/2}).$

• Assumption B4: The second derivative of $\tilde{G}_{n,f}$ exists and is given by $\tilde{G}^{\prime\prime}_{n,f}(\beta)=\frac{d^{2}}{d\beta^{2}}\tilde{G}_{n,f}(\beta)$. Moreover, letting $\tilde{r}^{\prime\prime}_{n,f}(\beta)=\frac{d^{2}}{d\beta^{2}}r_{n,f}(\beta)$, we have

  $\displaystyle\sup_{f\in\mathcal{F},\beta\in\mathcal{B}}|\tilde{r}^{\prime\prime}_{n,f}(\beta)|=o_{P}(1).$

Assumption B1 states that the estimator for the goodness-of-fit along any parametric sub-model takes the same value at $\beta=0$. In view of condition (7), all sub-models intersect and attain the same value for the goodness-of-fit at $\beta=0$, so it is natural to assume that our estimator also has this property. Assumptions B2 places a requirement that $\tilde{G}_{n,f}(\beta)$ is an asymptotically linear estimator for $G_{0,f}(\beta)$, where the asymptotic linearity holds uniformly over $\mathcal{F}\times\mathcal{B}$. Assumption B3 states that $\tilde{G}_{n,f}(\beta)$ is differentiable, and the derivative $\tilde{G}_{n,f}^{\prime}(\beta)$ is an asymptotically linear estimator for $G^{\prime}_{0,f}(\beta)$, uniformly over $\mathcal{F}\times\mathcal{B}$. Finally, Assumption B4 requires that the second derivative of $\tilde{G}_{n,f}(\beta)$ exists and converges in probability to the second derivative of $G^{\prime\prime}_{0,f}(\beta)$, uniformly over $\mathcal{F}\times\mathcal{B}$.

For a given estimator $\{\tilde{G}_{n,f}(\beta):f\in\mathcal{F},\beta\in\mathcal{B}\}$ of $\{G_{0,f}:f\in\mathcal{F},\beta\in\mathcal{B}\}$, let $\{\tilde{\beta}_{n,f}:f\in\mathcal{F}\}$ satisfy

The following theorem states that, under mild regularity conditions, $\tilde{\beta}_{n,f}$ is an asymptotically linear estimator for $\beta_{0,f}$, and moreover the collection $\{\tilde{\beta}_{n,f}:f\in\mathcal{F}\}$, when appropriately standardized, achieves a Gaussian limiting distribution.

Theorem 1 can be viewed as a generalization of well-known results that show M-estimators are asymptotically linear in finite-dimensional models (see, e.g., Theorem 5.23 of van der Vaart, 2000). Our result on uniform asymptotic linearity in infinite-dimensional models can be proven using a fairly standard argument.

In what follows, we suggest some approaches for constructing an estimator that satisfies Assumptions B1-B4. We describe at a high-level what types of conditions are needed for a given estimation strategy to be valid, though the specific requirements depend on the target estimand $G_{0,f}$ and vary from problem to problem. Later on in Section 7, we demonstrate how to construct an estimator and that satisfies Assumptions B1-B4 in an example.

Suppose that one has available an estimator $\hat{P}_{n}$ for the underlying probability distribution $P_{0}$. Typically estimation of the entire probability distribution $P_{0}$ it not necessary, and one will only need to estimate nuisance components upon which $G_{0,f}(\beta)$ and $\phi_{0,f}(\cdot;\beta)$ depend. We assume that $\{G_{P,f}(\beta):f\in\mathcal{F},\beta\in\mathcal{B}\}$ and $\{\phi_{P,f}(\cdot;\beta):f\in\mathcal{F},\beta\in\mathcal{B}\}$ depend on $P$ only through a nuisance $Q_{P}$, which resides in a space $\mathcal{Q}$ endowed with norm $\|\cdot\|_{\mathcal{Q}}$. The true value of the nuisance component is given by $Q_{P_{0}}$, and the plug-in estimator for the nuisance is $Q_{\hat{P}_{n}}$.

As a starting point, one might consider using $G_{\hat{P}_{n},f}(\beta)$ as an estimator for $G_{0,f}(\beta)$. If $\hat{P}_{n}$ belongs to the model $\mathcal{M}$, the plug-in estimator satisfies Assumption B1. This leaves Assumptions B2 through B4 to be verified. Suppose now that $Q_{P}$ is itself pathwise differentiable and can therefore be estimated at an $n^{1/2}$-rate. Then if $Q_{\hat{P}_{n}}$ is an asymptotically linear estimator for $Q_{P_{0}}$, one can argue that $G_{\hat{P}_{n},f}(\beta)$ is also be asymptotically linear by applying the delta method. Assumption B2 then holds, as long as the asymptotic linearity is preserved uniformly over $\mathcal{F}\times\mathcal{B}$. Asymptotic linearity of $G^{\prime}_{\hat{P}_{n},f}$ (Assumption B3) and consistency of $G^{\prime\prime}_{\hat{P}_{n},f}$ (Assumption B4) can be established using a similar argument.

In many instances, the nuisance $Q_{P}$ can include quantities such as density functions or conditional mean functions which are non-pathwise differentiable in a nonparametric model. In this case, it is not possible to construct an $n^{1/2}$-rate consistent estimator for the nuisance. Obtaining an estimator for the nuisance usually involves making a bias variance trade-off that may be sub-optimal for the objective of estimating the goodness-of-fit. When the nuisance estimator retains non-negligible bias, it is possible that the bias propagates, leading to $G_{\hat{P}_{n},f}(\beta)$ being biased as well. As a consequence, $G_{\hat{P}_{n},f}(\beta)$ may not be asymptotically linear, and we may require more sophisticated methods to construct an $n^{1/2}$-consistent estimator.

One widely-used method for obtaining an asymptotically linear estimator when the initial estimator $G_{\hat{P}_{n},f}(\beta)$ is biased is to perform a one-step bias correction (Pfanzagl, 1982). Consider the plug-in estimator for the efficient influence function $\phi_{\hat{P},f}(\cdot;\beta)$. The empirical average of the estimator for the efficient influence function serves as a first-order correction for the bias of the initial estimator. By adding this empirical average to the initial estimator, one can obtain the so-called one-step estimator:

It can be easily seen that the one-step estimator satisfies Assumption B1. In what follows, we briefly discuss what arguments one will typically use to verify Assumptions B2-B4. While we do not provide a detailed discussion here, we refer readers to a recent review by Hines et al. (2022), which provides a more in-depth explanation.

The estimation error of the one-step estimator has the exact representation,

where we define $R^{\mathrm{i}}_{n,f}(\beta)$ and $R^{\mathrm{ii}}_{n,f}(\beta)$ as

Asymptotic linearity of the one-step estimator follows if it can be established that $R^{\mathrm{i}}_{n,f}(\beta)$ and $R^{\mathrm{ii}}_{n,f}(\beta)$ converge to zero in probability at an $n^{1/2}$-rate. The first term $R^{\mathrm{i}}_{n,f}(\beta)$ is a difference-in-differences remainder that is asymptotically negligible when $\phi_{\hat{P}_{n},f}(\cdot;\beta)$ is consistent for $\phi_{P_{0},f}(\cdot;\beta)$ and $\phi_{\hat{P}_{n},f}(\cdot;\beta)$ is contained within a $P_{0}$-Donsker class (see Lemmas 19.24 and 19.26 of van der Vaart, 2000). The second term $R^{\mathrm{ii}}_{n,f}(\beta)$ is a second-order remainder term, which can usually be bounded above by the squared norm of the difference between the nuisance estimator and its true value, $\|Q_{\hat{P}_{n}}-Q_{P_{0}}\|_{\mathcal{Q}}^{2}$. One can argue that if the nuisance estimator is $n^{1/4}$-consistent with respect to $\|\cdot\|_{\mathcal{Q}}$, then $R^{\mathrm{ii}}_{n,f}(\beta)=o_{P}(n^{-1/2})$. Even in a nonparametric model, there exist several approaches for constructing $n^{1/4}$-rate consistent nuisance estimators when one makes only mild structural assumptions on $Q_{P_{0}}$, such as smoothness or monotonicity (see, e.g., van de Geer, 2000; Tsybakov, 2009). To verify that Assumptions B3 and B4 hold, one can perform a similar analysis to show that the first and second derivatives of the remainder terms $R^{\mathrm{i}}_{n,f}(\beta)$ and $R^{\mathrm{ii}}_{n,f}(\beta)$, with respect to $\beta$, tend to zero at the requisite rate.

While we focused on one-step estimation above because we find its simplicity appealing, other strategies for constructing of bias-corrected estimators, such as targeted minimum-loss based estimation could alternatively be used. These strategies are usually also viable under a similar set of regularity conditions.

We are at this point prepared to describe the bias-corrected estimator for $\Psi_{0}$ and its asymptotic properties. As stated in Section 3, we estimate $\Psi_{0}$ as

where $\{\tilde{G}_{n,f}(\beta):f\in\mathcal{F},\beta\in\mathcal{B}\}$ is an estimator satisfying the conditions outlined in Section 4.1.

In this section we establish weak convergence of $\tilde{\Psi}_{n}$. We show that $\tilde{\Psi}_{n}$ attains a tractable limiting distribution under mild regularity conditions, but the limiting distribution and convergence rate depend on the true value of $\Psi_{0}$. We study two cases. First, we consider the setting in which $\Psi_{0}=0$, and the null hypothesis of no improvement in fit (3) holds. Second, we study the case in which $\Psi_{0}$ is a positive constant.

Suppose that the null of no improvement in fit holds. Recall from Section 2 that when $\Psi_{0}=0$, $\sup_{f\in\mathcal{F}}|\beta_{0,f}|=0$. Also, as discussed in Section 3, by performing a Taylor expansion for $\tilde{G}_{n,f}$ around $\tilde{\beta}_{n,f}$, we have

for some $\check{\beta}_{n,f}$ satisfying $|\check{\beta}_{n,f}-\beta_{0,f}|\leq|\tilde{\beta}_{n,f}-\beta_{0,f}|$. Under Assumption B4, we are able, in (11), to replace $G^{\prime\prime}_{n,f}(\check{\beta}_{n,f})$ with $G^{\prime\prime}_{0,f}(\beta_{0,f})$. This and the fact that $\sup_{f\in\mathcal{F}}\beta_{n,f}^{2}=O_{P}(n^{-1})$ allow us to write

Thus, under the null, $\tilde{\Psi}_{n}$ can be represented as the squared supremum of an empirical process, plus an asymptotically negligible remainder. By applying Theorem 1 in conjunction with Slutsky’s theorem and the continuous mapping theorem, we have that $n\tilde{\Psi}_{n}$ converges weakly to $\sup_{f\in\mathcal{F}}\left[\left\{2G^{\prime\prime}_{0,f}(0)\right\}^{-1/2}\mathbb{H}(f)\right]^{2}$, where $\mathbb{H}$ is the Gaussian process described in Theorem 1. The following Theorem states this result formally.