Testing with p*-values: Between p-values, mid p-values, and e-values

Hypothesis testing is usually conducted with the classic notion of p-values. We introduce the abstract notion of p*-variables, with p*-values as their realizations, defined via a simple inequality in stochastic order, in a way similar to p-variables and p-values. As generalized p-variables, p*-variables are motivated by mid p-values, and they admit four natural interpretations: operational, probabilistic, Bayesian, and randomized testing, arising in various statistical contexts.

The most important and practical example of p*-values is the class of mid p-values (Lancaster [16]), arising from discrete test statistics. Mid p-variables are not p-variables, but they are p*-variables. Discrete test statistics appear in many applications, especially when data represent frequencies or counts; see e.g., Döhler et al. [8] in the context of false discovery rate control. Another example of discrete p-values is the conformal p-values (Vovk et al. [32]); see e.g., the recent study of Bates et al. [2] on detecting outliers using conformal p-values. Using mid p-values is one way to address discrete test statistics; another way is using randomized p-values. We refer to Habiger [14] for mid and randomized p-values in multiple testing, and Rubin-Delanchy et al. [23] for probability bounds on combining independent mid p-values based on convex order.

In addition to mid p-values, p*-values are also naturally connected to e-values. E-values have been recently introduced to the statistical community by Vovk and Wang [35], and they have several advantages in contrast to p-values, especially via their connections to Bayes factors and test martingales (Shafer et al. [28]), betting scores (Shafer [27]), universal inference (Wasserman et al. [38]), anytime-valid tests (Grünwald et al. [13]), conformal tests (Vovk [31]), and false discovery rate under dependence (Wang and Ramdas [37]).

In discussions where the probabilistic specification as random variables is not emphasized, we will loosely use the term “p/p*/e-values” for both p/p*/e-variables and their realizations, similarly to Vovk and Wang [34, 35], and this should be clear from the context.

The relationship between p*-values and mid p-values is studied in Section 3. We obtain a new stochastic representation for mid p-values (Theorem 3.1), which unifies the classes of p-, mid p-, and p*-values. The set of p*-values is closed under several types of operations, and this closure property is not shared by that of p-values or mid p-values (Proposition 3.3). Based on these results, we find that p*-values serve as an abstract and generalized version of mid p-values which is mathematically more convenient to work with.

There are several equivalent definitions of p*-variables: by stochastic order (Definition 2.2), by averaging p-variables (Theorem 4.1), by conditional probability (Proposition 4.3), and by randomized tests (Proposition 4.5); each of them represents a natural statistical path to a generalization of p-values, and these paths lead to the same mathematical object of p*-values. Moreover, a p*-value is a posterior predictive p-value of Meng [20] in the Bayesian context. The p*-test in Section 4.4 is a randomized version of the traditional p-test; the randomization is needed because p*-values are weaker than p-values.

Merging methods are useful in multiple hypothesis testing for both p-values and e-values. Merging several p-values or e-values are used, either directly or implicitly, in false discovery control procedures in Genovese and Wasserman [11] and Goeman and Solari [12] and generalized Bonferroni-Holm procedures (see [34] for p-values and [35] for e-values). We study merging functions for p*-values in Section 5, which turn out to have convenient structures. In particular, we find that a (randomly) weighted geometric average of arbitrary p*-variables multiplied by $\mathrm{e}\approx 2.718$ is a p-variable (Theorem 5.2), allowing for a simple combination of p*-values under unknown dependence; a similar merging function for p-values is obtained by [34]. In the setting of merging independent p*-values, inequalities obtained by [23] on mid p-values can be directly applied to p*-values.

We explore in Sections 6 the connections among p-values, p*-values, and e-values by establishing results for admissible calibrators. Figure 1 summarizes these calibrators where the ones between p-values and e-values are obtained in [35]. Notably, for an e-value $e$, $(2e)^{-1}$ is a calibrated p*-value, which has an extra factor of $1/2$ compared to the standard calibrated p-value $e^{-1}$. A composition of the e-to-p* calibration $p^{*}=(2e)^{-1}\wedge 1$ and the p*-to-p calibration $p=(2p^{*})\wedge 1$ leads to the unique admissible e-to-p calibration $p=e^{-1}\wedge 1$, thus showing that p*-values serve as a bridge between e-values and p-values.

In classic statistical settings where precise (uniform on $[0,1]$) p-values are available, p*-values may not be directly useful as their properties are weaker than p-values. Nevertheless, applying a p*-test in situations where precise p-values are unavailable leads to several improvements on classic methods for p-values and e-values. An application on testing with e-values is discussed and numerically illustrated in Section 7, and one on testing with discrete test statistics and mid p-values is presented in Section 8. (Another application is presented in Appendix B.) From these examples, we find that the tool of p*-values is useful even when one is primarily interested in p-values, mid p-values, or e-values.

The paper is written such that p-values, p*-values and e-values are treated as abstract measure-theoretical objects, following the setting of [34, 35]. Our null hypothesis is a generic and unspecified one, and it can be simple or composite; nevertheless, for the discussions of our results, it would be harmless to keep a simple hypothesis in mind as a primary example. All proofs are put in Appendix A and the randomized p*-test is discussed in Appendix B.

Following the setting of [35], we directly work with a fixed atomless probability space $(\Omega,\mathcal{A},\mathbb{P})$, where our (global) null hypothesis is set to be the singleton $\{\mathbb{P}\}$. As explained in Appendix D of [35], no generality is lost as all mathematical results (of the kind in this paper) are valid also for general composite hypotheses. We assume that $(\Omega,\mathcal{A},\mathbb{P})$ is rich enough so that we can find a uniform random variable independent of a given random vector as we wish. We first define stochastic orders, which will be used to formulate the main objects in the paper. All terms like “increasing” and “decreasing” are in the non-strict sense.

Recall that the defining property of a p-value, realized by a random variable $P$, is that $\mathbb{P}(P\leq\alpha)\leq\alpha$ for all $\alpha\in(0,1)$, meaning that the type-I error of rejecting $P$ based on $P\leq\alpha$ is at most $\alpha$ (see e.g., [35]). The above is equivalent to the statement that $P$ is first-order stochastically larger than a uniform random variable on $[0,1]$ (e.g., Section 1.A of [29]). Motivated by this simple observation, we define p-variables and e-variables, and our new concept, called p*-variables, via stochastic orders.

We allow both p-variables and p*-variables to take values above one, although such values are uninteresting, and one may safely truncate them at $1$; moreover, we allow an e-variable $E$ to take the value $\infty$ (but with probability $0$ under the null), which corresponds to a p-variable taking the value $0$ (also with probability $0$).

Since $\leq_{1}$ is stronger than $\leq_{2}$, a p-variable is also a p*-variable, but not vice versa. Due to the close proximity between p-variables and p*-variables, we often use $P$ for both of them; this should not create any confusion. We refer to p-values as realizations of p-variables, p*-values as those of p*-variables, and e-values as those of e-variables. By definition, both a p-variable and a p*-variable have a mean at least $1/2$.

The classic definition of an e-variable $E$ is via $\mathbb{E}[E]\leq 1$ and $E\geq 0$ a.s. ([35]). This is equivalent to our Definition 2.2 because

We choose to express our definition via stochastic orders to make an analogy among the three concepts, and stochastic orders will be a main technical tool for results in this paper.

Our main focus is the notion of p*-values, which will be motivated from five perspectives in Sections 3 and 4.

An important motivation for p*-values is the use of mid p-values and discrete test statistics. We first recall the usual practice to obtain p-values. Let $T$ be a test statistic which is a function of the observed data. Here, a smaller value of $T$ represents stronger evidence against the null hypothesis. The p-variable $P$ is usually computed from the conditional probability

where $F$ is the distribution of $T$, and $T^{\prime}$ is an independent copy of $T$; here and below, a copy of $T$ is a random variable identically distributed as $T$.

If $T$ has a continuous distribution, then the p-variable $P$ defined by (2) has a standard uniform distribution. If the test statistic $T$ is discrete, e.g., when testing a binomial model $\mathrm{Binomial}(n,\pi)$, $P$ is strictly first-order stochastically larger than a uniform random variable on $[0,1]$.

The discreteness of $T$ leads to a conservative p-value; in particular, $\mathbb{E}[P]>1/2$. One way to address this is to randomize the p-value to make it uniform on $[0,1]$; however, randomization is generally undesirable in testing. As the most natural alternative, mid p-values (Lancaster [16]) arise in the presence of discrete test statistics. For the test statistic $T$, the mid p-value is given by

where $F(t-)=\lim_{s\uparrow t}F(t)$ for $t\in\mathbb{R}$. Clearly, $P_{T}\leq F(T)$ and $\mathbb{E}[P_{T}]=1/2$. If $T$ is continuously distributed, then $P_{T}=F(T)$ is uniform on $[0,1]$. In case $T$ is discrete, $P_{T}$ is not a p-variable. In Figure 2 we present some examples of quantile functions of p-, mid p- and p*-variables.

Similarly to the case of p-variables in Definition 2.2, we formally define a mid p-variable as a random variable $P\geq_{1}P_{T}$ for some test statistic $T$. Often we have equality (i.e., $P=_{1}P_{T}$), and a strict inequality may appear due to, e.g., composite hypotheses, similarly to the case of p-variables or e-variables.

It is straightforward to verify that mid p-variables are p*-variables; see [23] where convex order is used in replace of our second-order stochastic dominance. The following theorem establishes a new unified stochastic representation for p-, mid p-, and p*-variables.

From Theorem 3.1, it is clear that mid p-variables are special cases of p*-variables but the converse is not true. As a direct consequence, all results later on p*-variables can be directly applied to mid p-values. The stochastic representations in Theorem 3.1 may not be directly useful in statistical inference; nevertheless they reveal a deep connection between mid p-values and p*-values, allowing us to analyze possible improvements of methods designed for p*-variables when applied to mid p-values, and vice versa.

In the next result, we summarize some closure property of the sets of p-, mid p and p*-variables.

Proposition 3.3 suggests that the set of p*-variables has the nicest closure properties among the three. Moreover, we will see in Theorem 4.1 below that the set of p*-variables is precisely the convex hull of the set of p-variables, and hence, it is also the convex hull of the set of mid p-variables which contains all p-variables.

In this section, we will present four further equivalent definitions of p*-values, each arising from a different statistical context, and providing several interpretations of p*-values.

Merging p-values and e-values is extensively studied in the literature of multiple hypothesis testing; see the recent studies [18, 35, 33] and the references therein. We will be interested in merging p*-values (including mid p-values) into both a p*-value and a p-value. The following proposition gives a convenient conversion rule between p*-values and p-values. The fact that two times a p*-variable is a p-variable is already observed by [20].

Proposition 5.1 implies that, in order to obtain a valid p-value from several p*-values, a naive method is to multiply each p*-value by $2$ and then apply a valid method for merging p-values (under the corresponding assumptions). We will see in the next few results that we can often obtain stronger results than this.

As argued by Efron [10, p.50-51], dependence assumptions are difficult to verify in multiple hypothesis testing. We will first focus on the case of arbitrarily dependent p*-values, that is, without making any assumptions on the dependence structure of the p*-variables, and then turn to the case of independent or positively dependent p*-variables.

In this section, we discuss calibration between p-, p*-, and e-values. Calibration between p-values and e-values is one of the main topics of [35].

In this section we discuss applications of p*-values to tests with e-values and martingales. E-values and test martingales are usually used for purposes more than rejecting a null hypothesis while controlling type-I error; in particular, they offer anytime validity and different interpretations of statistical evidence (e.g., [13]). We compare the power of several methods here for a better understanding of their performance, while keeping in mind that single-run detection power (which is maximized by p-values if they are available) is not the only purpose of e-values.

Suppose that $E$ is an e-variable, usually obtained from likelihood ratios or stopped test supermartingales (e.g., [28], [27]). A traditional e-test is

Using the fact that $(2E)^{-1}$ is a p*-variable in Theorem 6.2 (iii) , we can design the randomized test

where $V\sim\mathrm{U}[0,2\alpha]$ is independent of $E$ (Proposition 4.5). The test (13) has $3/4$ chance of being more power than the traditional choice of testing $E$ against $1/\alpha$ in (12). Randomization is undesirable, but (13) inspires us to look for alternative deterministic methods.

Suppose that one has two independent e-variables $E_{1}$ and $E_{2}$ for a null hypothesis. As shown by [35], it is optimal in a weak sense to use the combined e-variable $E_{1}E_{2}$ for testing the null. Assume further that one of $E_{1}$ and $E_{2}$ satisfies condition (DE).

Using (13) with the random threshold $\alpha E_{2}$ and Proposition B.5 in Appendix B, we get $\mathbb{P}((2E_{1})^{-1}\leq\alpha E_{2})\leq\alpha$ (note that the positions of $E_{1}$ and $E_{2}$ are symmetric here). Hence, the test

has size at most $\alpha$. The threshold of the test (14) is half the one obtained by directly applying (12) to the e-variable $E_{1}E_{2}$. Thus, the test statistic is boosted by a factor of $2$ via condition (DE) on either $E_{1}$ or $E_{2}$. No assumption is needed for the other e-variable. In particular, by setting $E_{2}=1$, we get a p-variable $(2E_{1})^{-1}$ if $E_{1}$ satisfies (DE), as we see in Proposition 6.4.

E-values calibrated from p-values are useful in the context of testing randomness online (see [31]) and designing test martingales (see [9]). More specifically, for a possibly infinite sequence of independent p-variables $(P_{t})_{t\in\mathbb{N}}$ and a sequence of p-to-e calibrators $(f_{t})_{t\in\mathbb{N}}$, the stochastic process

is a supermartingale (with respect to the filtration of $(P_{t})_{t\in\mathbb{N}}$) with initial value $X_{0}=0$ (it is a martingale if $P_{t}$, $t\in\mathbb{N}$ are standard uniform and $f_{t}$, $t\in\mathbb{N}$ are admissible). As a supermartingale, $(X_{t})_{t\in\mathbb{N}}$ satisfies the anytime validity, i.e., $X_{\tau}$ is an e-variable for any stopping time $\tau$; moreover, Ville’s inequality gives

The process $(X_{t})_{t\in\mathbb{N}}$ is called an e-process by [37]. Anytime validity is crucial in the design of online testing where evidence arrives sequentially in time, and scientific discovery is reported at a stopping time considered with sufficient evidence.

Notably, the most popular choice of p-to-e calibrators are those in (10) and (11) (see e.g., [31]), which are convex. Theorem 6.2 implies that if the inputs are not p-values but p*-values, we can still obtain e-processes using convex calibrators such as (10) and (11), without calibrating these p*-values to p-values. This observation becomes useful when each observed $P_{t}$ is only a p*-variable, e.g., a mid p-value or an average of several p-values from parallel experiments.

Moreover, for a fixed $t\in\mathbb{N}$, if there is a convex $f_{s}$ for some $s\in\{1,\dots,t\}$ with $f_{s}(1)=0$, and $P_{s}$ is a p-variable (the others can be p*-variables with any p*-to-e calibrators), then (DE) is satisfied by $f_{s}(P_{s})$, and we have $\mathbb{P}(X_{t}\geq 1/\alpha)\leq\alpha/2$ by using the test (14); see our numerical experiments below.

We compare by simulation the performance of a few tests via merging mid p-values. The (global) null hypothesis $H_{0}$ is that the test statistic $T$ follows a binomial distribution $\mathrm{Binomial}(n,\pi)$, and $K$ tests are conducted. We set $n=40$ and $\pi=0.3$, so that the obtained p-values are considerably discrete. We denote by $P_{1},\dots,P_{K}$ the obtained p-values via (2), and $P^{*}_{1},\dots,P^{*}_{K}$ the obtained mid p-values via (3). Let $\bar{P}^{*}$ and $\tilde{P}^{*}$ be the arithmetic average and the geometric average of $P^{*}_{1},\dots,P^{*}_{K}$, respectively.

The true data probability generating the test statistics is a binomial distribution $\mathrm{Binomial}(n,(1-\theta)\pi)$, where $\theta\in[0,1]$. The case $\theta=0$ means that the null hypothesis $H_{0}$ is true, and a larger $\theta$ indicates a stronger signal.

We allow the test statistics to be correlated, and this is achieved by simulating from a Gaussian copula with common pairwise correlation parameter $\rho$ (more precisely, we first simulate from a Gaussian copula, and then obtain the observable discrete test statistics by a quantile transform). We consider the following tests (there are other tests possible for this setting, and we only compare these four to illustrate a few relevant points):

1. (a)

   the probability bound (7) on the arithmetic mean in [23]: reject $H_{0}$ if $\bar{P}^{*}\leq 1/2-(-\log(\alpha)/(6K))^{1/2}$;

2. (b)

   the arithmetic mean times $2$ using Proposition 5.3: reject $H_{0}$ if $\bar{P}^{*}\leq\alpha/2$;

3. (c)

   the geometric average of p*-values using Theorem 5.2: reject $H_{0}$ if $\tilde{P}^{*}\leq\alpha/\mathrm{e}$;

4. (d)

   the Bonferroni correction: reject $H_{0}$ if $\min(P_{1},\dots,P_{K})\leq\alpha/K$.

Note that tests (a), (b) and (c) use mid p-values based on methods for p*-values, and (d) uses p-values. All of (b), (c) and (d) are valid tests under arbitrary dependence (AD) whereas the validity of (a) requires independence. Therefore, we expect (a) to perform very well in case independence holds. All other methods are valid but conservative, as there is a big price to pay to gain robustness against all dependence structures.

The significance level $\alpha$ is set to be $0.05$ for good visibility. The power of the four tests is reported in Figure 4 which is computed from the average of 10,000 replications for varying signal strength $\theta\in[0,1]$ and for $\rho=0$ (independence), $\rho=0.2$ (mild dependence) and $\rho=0.8$ (strong dependence). The situation of $\rho=0.8$ is the most relevant to us as averaging methods are designed mostly for situations where the presence of strong or complicated dependence is suspected.

As we can see from Figure 4, the test (a) relying on independence has the strongest power as expected. Its size becomes largely inflated as soon as mild dependence is present, and hence it can only be applied in situations where independence among obtained mid p-values can be justified. Indeed, the size of test (a) can be $\approx 0.4$ in case $\rho=0.2$, which is clearly not useful. Among the three methods that are valid for arbitrary dependence, the geometric average (c) has stronger power for large $\rho$ and the Bonferroni correction (d) has stronger power for small $\rho$. The arithmetic average (b) performs poorly unless p-values are very strongly correlated. These observations on merging mid p-values are consistent with those in [34] on merging p-values. In conclusion, the geometric average (c) can be useful when the dependence among mid p-values is speculated to be strong or complicated, although unknown to the decision maker.

In this paper we introduced p*-values (p*-variables) as an abstract measure-theoretic object. The notion of p*-values is a manifold generalization of p-values, and it enjoys many attractive theoretical properties in contrast to p-values. In particular, mid p-values, which arise in the presence of discrete test statistics, form an important subset of p*-values. Merging methods for p*-values are studied. In particular, a weighted geometric average of arbitrarily dependent p*-values multiplied by $\mathrm{e}\approx 2.718$ yields a valid p-value, which can be useful when multiple mid p-values are possibly strongly correlated.

Results on calibration between p*-values and e-values reveal that p*-values serve as an intermediate step in both the standard e-to-p and p-to-e calibrations. Although a direct test with p*-values may involve randomization, we find that p*-values are useful in the design of deterministic tests with averages of p-values, mid p-values, and e-values. From the results in this paper, the concept of p*-values serves a useful technical tool which enhances the extensive and growing applications of p-values, mid p-values, and e-values.

[Acknowledgments] The author thanks Ilmun Kim, Aaditya Ramdas, and Vladimir Vovk for constructive comments on an earlier version of the paper, and Tiantian Mao, Marcel Nutz, and Qinyu Wu for kind help on some technical statements.

The author acknowledges financial support from and the Natural Sciences and Engineering Research Council of Canada (RGPIN-2018-03823, RGPAS-2018-522590).