A Nonparametric Approach for Estimating the Effective Sample Size in Gaussian Approximation of Expected Value of Sample Information

The effective sample size (ESS) is a statistical concept that quantifies the informational value of a probability distribution through the number of hypothetical participants morita2008determining . A greater ESS signifies a more informative distribution, while a lower value indicates the opposite. The estimation of ESS serves two important purposes in health economic evaluation. Firstly, it allows area experts to determine if the amount of information contained in the distribution aligns with their prior beliefs. Secondly, it facilitates the estimation of the EVSI within the GA approach.jalal2018gaussian ; linke2023estimating This efficiency stems from the characteristic of ESS as being intrinsic to the GA setup, where it is computed only once. After that, this value remains unaltered for EVSI calculations across a spectrum of study designs, irrespective of sample size variations. ¹¹1To estimate EVSI, ESS is combined with the nonparametric regression models and Taylor series expansions to efficiently approximate the distribution of the conditional economic benefit and estimate EVSI. We refer readers to the article by Li et al. for a comprehensive explanation of the EVSI estimation method.linke2023estimating This section introduces the concept of ESS as it is presented within the GA approach.

Firstly, assume we have a parameter $\phi$ with a corresponding probability distribution $p(\phi)$ that can be well approximated by a Gaussian distribution. This distribution has mean $\mu_{0}$ and variance $\frac{\sigma^{2}}{n_{0}}$:

where $\sigma^{2}$ represents the individual-level variance of the data collection process, and $n_{0}$ can be understood as the number of hypothetical participants that would yield the uncertainty level of $\phi$. $n_{0}$ is defined as the ESS of $p(\phi)$ in the GA approach.

Additionally, we denote the set of $n$ observations obtained from the data collection exercise that would provide additional information on $\phi$ as $\bm{X}_{n}$, and its sample mean as $\widebar{\bm{X}}_{n}$. In the GA approach, it is assumed that given the parameter $\phi$, $\widebar{\bm{X}}_{n}$ is approximately Gaussian distributed with mean $\phi$ and variance $\frac{\sigma^{2}}{n}$:

Due to the conjugate nature of the distributions of $\phi$ and $\widebar{\bm{X}}_{n}|\phi$, the conditional mean of $\phi$ given the observed data is a weighted combination of $\mu_{0}$ and $\widebar{\bm{X}}_{n}$:

Using this formulation, the ESS $n_{0}$ can be determined based on the variance of either $\mathbb{E}_{\phi}[\phi|\bm{X}_{n}]$ or $\widebar{\bm{X}}_{n}$. This is because the marginal distributions of $\mathbb{E}_{\phi}[\phi|\bm{X}_{n}]$ and $\widebar{\bm{X}}_{n}$ are both Gaussian with variances $\frac{v\sigma^{2}}{n_{0}}$ and $\frac{\sigma^{2}}{vn_{0}}$, respectively. jalal2018gaussian As the variance of $p(\phi)$ is $\frac{\sigma^{2}}{n_{0}}$, we can estimate $n_{0}$ using variance ratios. Specifically, the variance ratio between $\phi$ and $\mathbb{E}_{\phi}[\phi|\bm{X}_{n}]$ yeilds an $n_{0}$ estimation of:

and the variance ratio between $\phi$ and $\widebar{\bm{X}}_{n}$ yeilds an $n_{0}$ estimation of:

Although equations 4 and 5 are derived using the Gaussian assumption, Jalal and Alarid-Escudero have shown that they can be generalized to estimate $n_{0}$ for other non-Gaussian likelihood and prior combinations. jalal2018gaussian

In this section, we review current $n_{0}$ estimation methods and note their drawbacks.jalal2018gaussian We then introduce a novel approach, which employs nonparametric regression models, to efficiently and accurately estimate $n_{0}$.

In the original GA article, Jalal and Alarid-Escudero introduced two methods that can be generalized to estimate $n_{0}$ for nonconjugate priors. jalal2018gaussian The first method uses equation 4. For this, we need to estimate $\mathbb{E}_{\phi}[\phi|\bm{X}_{n}]$ for a large number of datasets $\bm{X}_{n}$ generated from a specific data collection exercise. For each dataset, we can use Markov Chain Monte Methods (MCMC) to simulate from the posterior distribution of parameter $\bm{\phi}$ and calculate the mean. We then compute the sample variance of these estimated posterior means to approximate $\mbox{Var}_{\bm{X}_{n}}\left[\mathbb{E}_{\phi}[\phi|\bm{X}_{n}]\right]$ and estimate $n_{0}$. While this method yields accurate $n_{0}$ estimates, repeated MCMC evaluations may impose a significant computational burden and complicate practical implementation.

The second method uses equation 5. However, since the sample mean $\widebar{\bm{X}}_{n}$ may not have the same scale as $\phi$ for many non-Gaussian probability distributions, equation 5 can not be directly applied. In such cases, we need to identify other summary statistics $S$ that share the same scale as $\phi$ and compare its variance with that of $\phi$ to estimate $n_{0}$:

As this method only requires summary statistics, it is more computationally efficient than MCMC. However, it necessitates the identification of suitable summary statistics for $\bm{X}_{n}$. These statistics should exhibit the same scale as $\phi$ and have an appropriate level of variation, which can pose challenges in real-world case studies.

To reduce the computational burden of $n_{0}$ when the appropriate summary statistics of $\phi$ are difficult to find, we suggest estimating $\mathbb{E}_{\phi}[\phi|\bm{X}_{n}]$ using the nonparametric regression-based method, motivated by the EVSI method proposed by Strong et al. strong2015estimating ; strong2014estimating To achieve this, we need to first determine a low-dimensional summary statistic of the dataset $\bm{X}_{n}$, denoted $T(\bm{X}_{n})$. Note that, different from the current summary statistics-based method, the regression-based method does not require $T(\bm{X}_{n})$ to be converted to the same scale as $\phi$, which makes it simpler to specify. A commonly used summary statistic for the nonparametric regression-based method is the maximum likelihood estimate (MLE) of $\phi$ given $X_{n}$.

After we find $T(\bm{X}_{n})$, using the formulation suggested by Strong et al., $\phi$ can be expressed as the function of $T(\bm{X}_{n})$, plus a zero mean error term $\epsilon$ strong2015estimating

This equation suggests that the conditional expectation $\mathbb{E}_{\phi}[\phi|\bm{X}_{n}]$ can be approximated by the fitted value of a nonparametric regression model which takes $\phi$ as the response and $T(\bm{X}_{n})$ as the predictor. Therefore, our proposed method begins by fitting this non-parametric regression and then extracting the fitted values from this regression model. Next, $\mbox{Var}_{\bm{X}_{n}}\left[\mathbb{E}_{\phi}[\phi|\bm{X}_{n}]\right]$ is approximated by the variance of these fitted values. Finally, $n_{0}$ can then be estimated from equation 5. The detailed algorithm for estimating $n_{0}$ using the nonparametric regression-based method is summarized in box 1.

We assess the accuracy of the proposed method across seven diverse data collection scenarios, including settings where $\phi$ is univariate and multivariate. The distributions chosen for prior and data generation, respectively, for seven scenarios are: 1) beta-binomial, 2) gamma-exponential, 3) Poisson-gamma, 4) Gaussian-Weibull, 5) Dirichlet-multinomial, 6) truncated normal-binomial and 7) transformed beta-exponential. For each scenario, we compute $n_{0}$ using the original summary statistic and MCMC methods, and our nonparametric regression-based method. Moreover, for scenarios one to four, where closed-form posterior distributions exist, we derive the analytic value of $n_{0}$. Table 1 details the prior distributions, likelihood functions, and $n_{0}$ for each scenario.

To calculate the ESS using our nonparametric regression-based method, we generate $10^{4}$ parameter samples from each prior distribution. Each parameter sample is then inputted into the likelihood function of the study design to generate $10^{4}$ datasets. Subsequently, we compute the maximum likelihood estimate (MLE) as the summary statistic for each dataset. For our method, we then regressed the parameter samples on the MLEs using a spline. The fitted values were extracted from the regression model and used to compute $n_{0}$ according to equation 5. The summary statistics-based method also used the MLEs to estimate $n_{0}$ by equation 6, while the MCMC method was based on $10^{4}$ posterior samples for each dataset to estimate $\mathbb{E}_{\phi}[\phi|\bm{X}_{n}]$.jalal2018gaussian