Signal significance incorporating systematic uncertainty for continuous test

Scientists often lay claim to new discoveries, yet quantifying the extent of their deviation from established knowledge remains a persistent challenge. In last decades, the concept of “significance” has gained widespread adoption in scientific disciplines, particularly in physics, economics, biology, psychology, sociology, and medicine. It serves as a means to gauge the confidence with which a scientist asserts a new discovery. For instance, in high-energy physics, a conventional “rule” for significance is based on standard deviations ($\sigma$), where a signal with a significance of $3\sigma$ is considered as evidence, and only one with a significance of $5\sigma$ is deemed an observation of note. The paper [1] by P. K. Sinervo provides a historical review of the concept of “statistical significance” in observations and its application in high-energy experiments. Accurate estimation of significance is crucial in data analysis, as overestimation can damage one’s reputation, while underestimation may lead to overlooking important discoveries. A “significant” observation typically allows for the elimination of one or more hypotheses in favor of alternatives within a statistical framework. Hence, in essence, this constitutes a statistical problem, which is why significance is sometimes referred to as “statistical significance”. Two fundamental methods complement the calculation of significance: the counting method and the continuous test method. This paper follows the definition and calculation of statistical significance proposed in Ref. [2], which establishes a correlation between the normal distribution integral probability and the observed p-value. This approach yields explicit expressions for both counting experiments and continuous test statistics. For counting experiments,

$$\int^{S}_{-S}N(0,1)=1-p(n_{obs})=\sum^{n_{obs}-1}_{n=0}\frac{b^{n}}{n!}e^{-b}\ ,$$

(1)

where $S$, $n_{obs}$, $b$, and $N(0,1)$ are the signal significance, number of observed events, number of backgrounds, and normal distribution, respectively. For continuous test,

$$\int^{S}_{-S}N(0,1)=1-p(t_{obs})=\int^{t_{obs}}_{0}\chi^{2}(t;r)dt\ ,$$

(2)

where $t_{obs}\equiv 2[\mathrm{ln}L_{max}(s+b)-\mathrm{ln}L_{max}(b)]$ with $L$ is the likelihood obtained from fits, $L_{max}(s+b)$ is the maximum likelihood value with both signal and background in the fit and $L_{max}(b)$ is the maximum likelihood value with background in the fit. The $r$ is the difference in numbers of freedom.

The significance calculation may initially appear to be a mere statistical exercise, involving the numbers of observed events, backgrounds, and signals, as well as the uncertainties associated with these figures. However, it became evident that this calculation must extend beyond pure statistical analysis, as there are always systematic uncertainties related to factors such as the number of backgrounds, efficiency, signal shape, and background shape. These additional uncertainties introduce new complexities and difficulties to the significance calculation. In the case of the counting method, some proposed discussions on how to account for systematic uncertainty in significance calculation have been put forward [3; 4]. The central idea in these papers involves varying the conditions during the calculation based on the systematic uncertainty, and then selecting the “worst” (i.e., the least) significance as the final result. However, to the best of our knowledge, the consideration of systematic uncertainty in the continuous test method remains unaddressed. Therefore, this paper aims to explore the impact of systematic uncertainties on signal significance under various typical conditions and seeks to provide a solution for properly estimating signal significance when employing the continuous test method in the presence of systematic uncertainty.

We determine the signal significance using a continuous test approach with a toy model constructed from ad hoc invariant mass distributions. In this model, the data consists of two components, namely signal and background, and is generated with specific numbers and distributions. We assess the impact of three types of systematic uncertainties related to the background shape, number of backgrounds, and signal shape.

To begin, we generate a one-variable sample comprising 1500 events using the following formula

$$F(x)=(1-f)\cdot G(m,\sigma;x)+f\cdot P(a_{0},a_{1};x)\ ,$$

(3)

where $G(m,\sigma;x)$ and $P(a_{0},a_{1};x)$ are Gaussian and Polynomial functions, and $f$ is the ratio of the background. The values of the corresponding parameters are listed in Table 1. This sample will be used in the following for further studies. A fit to the generated sample, with $F(x)$ described by Eq. 3, is presented in Fig 1. And the statistical significance is estimated to be $6.7~{}\sigma$ by the continuous test method that is described by Eq. 2.

It would be intriguing to compare the effectiveness of the continuous test method and the counting method in estimating significance. Intuitively, the continuous test should offer advantages when the signal shape significantly differs from the background shape, as it provides additional information from the shapes. However, if the signal shape closely resembles that of the background, separating the signal and background from the fit would be challenging, making the counting method a better choice. To test this idea, we generated a series of samples with varying signal widths, i.e., $\sigma$ ranging from $5.0$ to $14.0$ in increments of $0.1$, and calculated the significance using the two methods described by Eqs. 1 and 2. The results, depicted in Fig. 5, confirm our expectations.

After investigating the impact of various systematic uncertainties on significance, our conclusion is that these uncertainties typically lead to unpredictable effects on signal significance. Due to the complex correlation between cause and result, we recommend a conservative approach: estimate signal significance by conducting trials with all possible combinations of uncertainties associated with the fitting procedure, and then select the “worst” outcome as the final result. Additionally, our study suggests that when the signal shape is distinguishable from the background, the continuous test method should be used for calculating signal significance, whereas the counting method is favored when the signal shape closes to the background. It’s important to note that this study solely focuses on significance calculation for signals of known location, without considering the look-elsewhere effect, which is beyond the scope of this paper and may require specific extensive methods for further exploration.